Full text of "Partnership for advanced computational infrastructure program : hearing before the Subcommittee on Basic Research of the Committee on Science, U.S. House of Representatives, One Hundred Fourth Congress, second session, March 19, 1996"

\A PARTNERSHIP FOR ADVANCED COMPUTATIONAL
INFMSTRUCTURE PROGRAM

Y4,SCI 2:104/47

Partnership for ftdvinced Conputatio. . .

jijiaRING

BEFORE THE

SUBCOMMITTEE ON BASIC RESEAKCH

OF THE

COMMITTEE ON SCIENCE
U.S. HOUSE OF REPRESENTATIVES

ONE HUNDRED FOURTH CONGRESS

SECOND SESSION

MARCH 19, 1996

[No. 47]

Printed for the use of the Committee on Science

OEPaSffORY '

SEP 1 8 1996

BnsrOWP«BUCLlBBAP'

U.S. GOVERNMENT PRINTING OFFICE
26-018 CC WASHINGTON : 1996

For sale by the U.S. Government Printing Office

Superintendent of Documents, Congressional Sales Office, Washington, DC 20402

ISBN 0-16-052912-3

PARTNERSHIP FOR ADVANCED COMPUTATIONAL
INFRASTRUCTURE PROGRAM

Y 4. SCI 2:104/47

Partnership for Advanced Conputatio .. .

nr^ARING

BEFORE THE

SUBCOMMITTEE ON BASIC RESEAKCH

OF THE

COMMITTEE ON SCIENCE
U.S. HOUSE OP REPRESENTATIVES

ONE HUNDRED FOURTH CONGRESS
SECOND SESSION

MARCH 19, 1996

[No. 47]

Printed for the use of the Committee on Science

SUPEBIfiTfNDEKTOFOOCOriL
DEWSiTORy

SEP 1 8 1996

BOSTON P«8UCLIBBAPV

U.S. GOVERNMENT PRINTING OFFICE
26-018 CC WASHINGTON : 1996

For sale by the U.S. Government Printing Office

Superintendent of Document.s. Congressional Sales Office. Washington. DC 20402

ISBN 0-16-052912-3

COMMITTEE ON SCIENCE

ROBERT S. WALKER,
F. JAMES SENSENBRENNER, Jr.,

Wisconsin
SHERWOOD L. BOEHLERT, New York
HARRIS W. FAWELL, Illinois
CONSTANCE A. MORELLA, Maryland
CURT WELDON, Pennsylvania
DANA ROHRABACHER, California
STEVEN H. SCHIFF, New Mexico
JOE BARTON, Texas
KEN CALVERT, California
BILL BAKER, California
ROSCOE G. BARTLETT, Maryland
VERNON J. EHLERS, Michigan**
ZACH WAMP, Tennessee
DAVE WELDON, Florida
LINDSEY 0. GRAHAM, South Carolina
MATT SALMON, Arizona
THOMAS M. DAVIS, Virginia
STEVE STOCKMAN, Texas
GIL GUTKNECHT, Minnesota
ANDREA H. SEASTRAND, California
TODD TL^RT, Kansas
STEVE LARGENT, Oklahoma
VAN HILLEARY, Tennessee
BARBARA CUBIN, Wyoming
MARK ADAM FOLEY, Florida
SUE MYRICK, North Carolina

Pennsylvania, Chairman
GEORGE E. BROWN, Jr., California RMM*
HAROLD L. VOLKMER, Missouri
RALPH M. HALL, Texas
BART GORDON, Tennessee
JAMES A. TRAFICANT, Jr., Ohio
JOHN S. TANNER, Tennessee
TIM ROEMER, Indiana
ROBERT E. (Bud) CRAMER, Jr., Alabama
JAMES A. BARCL\., Michigan
PAUL McHALE, Pennsylvania
JANE HARMAN, California
EDDIE BERNICE JOHNSON, Texas
DAVID MINGE, Minnesota
JOHN W. OLVER, Massachusetts
ALCEE L. HASTINGS, Florida
LYNN N. RIVERS, Michigan
KAREN McCarthy, Missouri
MIKE WARD, Kentucky
ZOE LOFGREN, California
LLOYD DOGGETT, Texas
MICHAEL F. DOYLE, Pennsylvania
SHEILA JACKSON LEE, Texas
WILLIAM P. LUTHER, Minnesota

David D. Clement, Chief of Staff and Chief Counsel

Barry Beringer, General Counsel

TiSH Schwartz, Chief Clerk and Administrator

Robert E. Palmer, Democratic Staff Director

Subcommittee on Basic Research

STEVEN SCHIFF, New Mexico, Chairman

JOE BARTON, Texas

BILL BAKER, California

VERNON J. Ehlers, Michigan

GIL GUTKNECHT, Minnesota

CONSTANCE A. MORELLA, Maryland

CURT WELDON, Pennsylvania

ROSCOE G. BARTLETT, Maryland

ZACH WAMP, Tennessee

DAVE WELDON, Florida

LINDSEY O. GRAHAM, South Carolina

VAN HILLEARY, Tennessee

SUE MYRICK, North Carolina

ROBERT E. (Bud) CRAMER, JR., Alabama
ALCEE L. HASTINGS, Florida
LYNN N. RIVERS, Michigan
LLOYD DOGGETT, Texas
WILLIAM P. LUTHER, Minnesota
JOHN W. OLVER, Massachusetts
ZOE LOFGREN, California
MICHAEL F. DOYLE, Pennsylvania
SHEILA JACKSON LEE, Texas
HAROLD L. VOLKMER, Missouri
BART GORDON, Tennessee

*Ranking Minority Member
**Vice Chairman

(II)

CONTENTS

WITNESSES

Page

March 19, 1996:

Dr. Paul Young, Assistant Director for CISE, National Science Founda-
tion, Arlington, Virginia 7

Dr. Edward Hayes, Chairman, Report on the Task Force on the Future
of NSF Supercomputing Centers Program, and Vice President for Re-
search, Ohio State University, Columbus, Ohio 26

Dr. Malvin Kalos, Director, Cornell Theory Center, Ithaca, New York 102

Dr. Larry Smarr, Director, National Center for Supercomputing Applica-
tions at UIUC (NCSA), Champaign, Illinois 107

Dr. Michael Levine and Dr. Ralph Roskies, Scientific Directors, Pitts-
burgh Supercomputing Center, Pittsburgh, Pennsylvania 168

Dr. Douglas Pewitt, Acting Director, San Diego Supercomputing Center,
La Jolla, California 169

Dr. Mary Vernon, Department of Computer Sciences and Engineering,
University of Washington, Seattle, Washington 188

Dr. Kelvin Droegemeier, CAPS Director, University of Oklahoma, Nor-
man, Oklahoma 192

Dr. Douglas Gale, Assistant Vice President of Information Systems and
Services, George Washington University, Washington, D.C 198

APPENDIX

Opening Statements:

Congressman Steve Schiff, Chairman, Subcommittee on Basic Research ... 204
Congressman Robert E. (Bud) Cramer, Jr., Ranking Minority Member,

Subcommittee on Basic Research 205

Congressman Michael F. Doyle 207

Revised statement for the record submitted by Dr. Ralph Roskies, Scientific

Director, Pittsburgh Supercomputing Center 207

Answers to questions submitted to Dr. Paul Young, Assistant Director for
CISE, National Science Foundation, by Members of the Subcommittee on
Basic Research 209

(III)

PARTNERSHIP FOR ADVANCED COMPUTA-
TIONAL INFRASTRUCTURE PROGRAM

TUESDAY, MARCH 19, 1996

U.S. House of Representatives,

Committee on Science,
Subcommittee on Basic Research,

Washington, DC.

The Subcommittee met at 1 p.m. in Room 2318 of the Rayburn
House Office Building, the Honorable Steven H. Schiff, Chairman
of the Subcommittee, presiding.

Mr. Schiff. The Subcommittee will please come to order.

Today the Subcommittee will exercise its oversight responsibil-
ities for the National Science Foundation by receiving testimony on
the solicitation of the Partnership for Advanced Computational In-
frastructure Program.

As this Congress continues to find new ways to balance the budg-
et and reduce the size of government, agencies are looking to maxi-
mize the value of each taxpayer dollar.

Of the government agencies, the National Science Foundation is
one of the best Federal agencies at running a lean and efficient or-
ganization. I extend my compliments to Dr. Lane and all of his em-
ployees.

This Congress provided the NSF with an increase in FY 1996
funding over FY 1995, but hard choices must still be made on fund-
ing priorities and the direction programs should be taking as we
head into the 21st Century.

One of those programs undergoing review is NSF's
Supercomputer Center program. NSF is making tough decisions on
the direction of the Supercomputer Centers, their funding, and
what role they should play in research applications.

The Partnership for Advanced Computational Infrastructure Pro-
gram builds on and replaces the current NSF Supercomputer Cen-
ters program established in 1985. The new program will focus on
taking advantage of newly emerging opportunities in high-perform-
ance computing and communications.

The Subcommittee is interested in assessing the major accom-
plishments of the NSF's Supercomputing Centers program over the
last 10 years, as well as the proposed restructuring of the program.

Additionally, the subcommittee desires to receive any comments
or specific suggestions from witnesses on the recompetition process
which NSF has developed for the program.

To outline a brief history of the Supercomputer program, the
NSF Supercomputing Centers Program was established in 1984 fol-

(1)

lowing strong expressions of the need for such computing resources
for the academic research community.

During a review in 1990, there was a distinct effort to expand
outreach services with initial efforts intended to forge closer ties to
industries that could profit from exposure to high-performance
computing and to include the community at large.

The Task Force Report we will receive testimony on today is the
latest of many studies of the Centers. The Director of the National
Science Foundation established the Task Force on the Future of
the NSF Supercomputer Centers Program in December 1994.

The Task Force was asked to analyze various alternatives for the
continuation, restructuring, or phaseout of NSF's current
Supercomputing Centers Program or the development of similar fu-
ture programs, and to make recommendations among the alter-
natives.

In January the National Science Board adopted the recommenda-
tion for a new competition of the Centers, and NSF announced the
Partnership for Advanced Computational Infrastructure.

Preliminary solicitations are due April 15th, 1996. The final
awards will be announced in the fall of 1996.

To learn more about the partnership program, it is a pleasure to
welcome Dr. Paul Young, the Assistant Director for Computer and
Information Science and Engineering at the National Science Foun-
dation.

We also welcome Dr. Ed Hayes, Chairman of the Task Force on
the Future of NSF Supercomputing Centers Program who rec-
ommended the National Science Board have a recompetition of the
existing Supercomputer Centers.

For our second panel we will welcome the directors of the four
Centers. Dr. Malvin Kalos of the Cornell Theory Center; Dr. Larry
Smarr of the National Center for Computing Applications; Dr.
Douglas Pewitt of the San Diego Supercomputing Center; and from
the Pittsburgh Supercomputing Center we have Dr. Michael Levine
and Dr. Ralph Roskies.

On our third panel we will hear from supercomputer researchers
Dr. Mary Vernon and Dr. Kelvin Droegemeir. We will hear also
from Dr. Douglas Gale who will present the views of organizations
which want to be involved in the new supercomputing program at
NSF.

Before calling the first panel up to testify, I first want to wel-
come officially here at the Subcommittee — I do not know that we
have done it on the record in the Full Committee yet — but welcome
officially to the Subcommittee my new Ranking Member from Ala-
bama, Congressman Cramer, and you are recognized for whatever
remarks you might like to make at this time.

Mr. Cramer. Thank you, Mr. Chairman. I will look forward to
working with you and the Committee Members both on my side of
the aisle and your side of the aisle, as well.

I am pleased to join the chairman in welcoming our distin-
guished witnesses this afternoon. I congratulate him on calling the
hearing on this important subject matter, the Future of the Na-
tional Science Foundation's Supercomputer Centers.

Today we are reviewing an NSF program which this subcommit-
tee has supported and encouraged since its inception ten years ago.

Hearings in the early 1980s highlighted concerns from the aca-
demic research community that researchers had very limited access
to advanced scientific computers. This was of concern because high-
performance computers were beginning to show enormous promise
as tools for attacking previously intractable problems. They had
opened entirely new areas of scientific inquiry.

At the outset, the Supercomputer Centers Program simply pro-
vided computer time to academic researchers at a few existing
supercomputer sites. That is not the way it works anymore.

The program was subsequently enlarged in scope to accelerate
the development and use of new hardware and software for science
and engineering applications, and to expand the number of re-
searchers skilled in the use of advanced computing technologies.

The NSF Supercomputer Centers have now become important in-
tellectual centers driving the rapid progress of scientific computing.

The impact of this program is summed up very well in a recent
report of the Task Force on the Future of the NSF Supercomputer
Centers Program.

Thorough review and assessment of the Supercomputer Centers
Program after 10 years is both reasonable and expected. NSF has
instituted two major external reviews over the past four years pro-
viding many opportunities for the affected research community to
express their views.

The restructuring plan for the Centers program, which the sub-
committee will hear more about today, has resulted from the rec-
ommendations of the Hayes Task Force. I am pleased that Dr.
Hayes is present to discuss them.

I would like to call attention to one particular recommendation
of this Hayes Task Force, which was. That the Leading Edge Com-
puter Center should be partnered with Regional Computer Centers
for maximum benefit to the academic research community.

We have such a regional computer center in my Congressional
District in Huntsville, Alabama, and I would like to submit for the
record a written statement from the Alabama Supercomputer Au-
thority, which indicates some of the activities of the Center and in-
dicates how regional centers can contribute to the new NSF Part-
nerships Program.

In light of the past performance of the Supercomputer Centers,
any proposal for restructuring the program must lay down a con-
vincing set of arguments that change will be beneficial.

This is particularly true since there seems to be agreement that
a principal role of the program remains unchanged. That is, to pro-
vide access to high-end computing infrastructure for the academic
science and engineering community.

I look forward to review of this Supercomputer Centers Program,
and in particular to a discussion of how the program may be
strengthened and how the proposed changes will affect the re-
search community, which has come to rely on this important source
for the conduct of science and engineering research.

I look forward to the subcommittee hearing today.

Mr. SCHIFF. Thank you, Mr. Cramer.

I want, before recognizing other Members, I just want to say
briefly that I want to affirm to you publicly what I have said pri-
vately.

During the six years I served on the Science Committee under
the Democratic Majority, I always felt treated very fairly by the
committee chairman, and by the subcommittee chairs, and I always
felt that partisanship never entered into this Committee, and that
if we had disagreements they were on the merits as we saw mat-
ters, and I want to assure you of my best intention to continue that
precedent.

With that

Mr. Cramer. Thank you.

Mr. SCHIFF. I note that we are joined by our Ranking Democratic
Member, Congressman Brown from California, who I recognize for
as much time as he wishes to consume.

[Bells ring.]

Mr. Brown. Thank you, Mr. Chairman. I will not consume a
great deal of time

[Bells begin to ring again.]

[Laughter.]

Mr. Brown. Uh-oh.

Mr. SCHIFF. I think that is a recess.

Mr. Brown. Two weeks ago there was a weekend program to
wire up as many schools as possible in California to the Internet,
which I thought was a marvelous PR thing, and actually did hook
up quite a few schools.

Yesterday morning I visited one of those schools and talked to an
8th grade class, or rather watched them operating their computers
and accessing the Worldwide Web and doing a lot of other things
that I thought were incomprehensible, or probably would not ever
have occurred.

This morning I met with the representatives of most of the major
cable companies in California.

Now I cite these three things to illustrate the changes that have
taken place in the relatively recent past.

Many of these changes I think can be attributed to this program
which has developed the capability for making better use of com-
puters in our institutions of higher learning, as well as our institu-
tions of lower learning.

It is one of the marvels of my experience here in Congress to
have been a part of this development over these last 10 or 15 years.

The next 10 or 15 years can be equally exciting. I won't be here
for most of that time, but many of you will and I urge you to take
the same care and concern over the further development of these
capabilities that have proven to be such a successful activity, and
one in which the committee has had a fairly important role over
the last 10 or 15 years.

It can give you great satisfaction in your work, I can assure you.

Thank you, Mr. Chairman.

Mr. SCHIFF. Thank you, Mr. Brown.

Does any other Member seek recognition for any opening state-
ment?

Mr. Ehlers?

Mr. Ehlers. Thank you, Mr. Chairman. Just briefly, several
items.

First of all, I am very impressed with the panel you have assem-
bled and I look forward to the hearing. I apologize that I have to

go to the Floor at some point to defend a bill that is up for consid-
eration.

The Chairman of the House Oversight Committee is tied up in
another committee, so I have to go there to defend it, and I apolo-
gize to the panel when I am absent.

The other point I would make is. A few weeks ago I circulated
a letter among my colleagues in the matter and in just a few days
received 89 signatures asking that we immediately put in place
full-year funding for the National Science Foundation.

I was assured that this would receive serious consideration, and
in fact it did, but it has been held up because there is an effort to
provide full-year funding for every part of the government that is
not fully funded at this point.

So that issue is in abeyance.

I simply want to get this on the record. Once again, if for some
reason we are not able to complete work on the Fiscal 1986 budget
soon, I believe it imperative that we provide full-year funding for
the National Science Foundation as soon as possible, since they op-
erate largely through grants, and many grants are being held up
pending completion of the budget, and many of the individuals in
this room, plus thousands of others throughout the country, are
anxiously awaiting confirmation of their grguits and final award of
the grants.

I think it is extremely important for us to recognize as a Con-
gress and as a Nation the important role that science plays in the
infrastructure for our economic engine. What we are about to hear
today says the same thing.

The Ranking Member of the Full Committee, Mr. Brown, for
years has been a strong advocate not only of science but particu-
larly of the supercomputing centers, as well as coordination of com-
puting activities among various agencies.

We are trying to carry on that mandate, and it has become ex-
tremely difficult because of the uncertainty of funding.

Now, Mr. Chairman, I believe it is very important that we get
the funding issue settled as soon as possible and ensure that we
continue as a Nation to have the computing capability we need to
carry on the scientific enterprise in this Nation.

I thank you for holding the hearing, and thank you especially for
assembling the panel that you have.

Thank you.

Mr. SCHIFF. Thank you, Mr. Ehlers.

As one of many of the dozens of lawyers in Congress, I am
pleased to have a real scientist here on the subcommittee.

Before I recognize more Members, I want to add two things very
briefly. The first is. I join you in the belief that if we are not able
to fund all agencies for the balance of the fiscal year, that we
should identify and fund the National Science Foundation sepa-
rately and would support that, and I appreciate your leadership in
it.

The second is, as you mentioned you have to go to the House
Floor in a bit to speak on behalf of a bill, so do I. I want to express
to those of you who are here who may never have seen a Congres-
sional hearing before, those of you who have been here before are

quite accustomed to the fact that we schedule several things at any
one time.

Those of you who have never been to a hearing before I under-
stand can be a little disconcerted at our coming and going, but
what I want to emphasize is that the main purpose of a hearing
is to record the information so in due course it is made available
to all Members of Congress.

And of course the most important person at the hearing, there-
fore, is the lady who is taking down everything that we are saying
back and forth because, regardless of who is sitting in this chair
or the other chairs, all that information gets across to the entire
Congress.

So I want to assure you of the importance we place on this hear-
ing, even if at times we may have to personally leave it for a bit.

Let me now recognize the gentleman from Pennsylvania, Con-
gressman Doyle.

Mr. Doyle. Thank you, very much.

Mr. Chairman, I want to thank you for holding today's hearing
on the National Science Foundation's Supercomputing Centers Pro-
gram.

As a Member who is fortunate enough to have one of these cen-
ters in my District, I am well aware of the contribution that they
have made and will continue to make to our Nation's technological
infrastructure.

I do want to start by welcoming our witness from the Pittsburgh
Center, Dr. Ralph Roskies. He, along with many other dedicated in-
dividuals, has made the Pittsburgh Supercomputing Center an
internationally recognized success.

The PSC has made meaningful contributions in many areas-no-
tably in assisting NIH research, developing weather models for the
National Weather Service, and helping EPA with its Air Quality
Assessments.

Today's hearing examines the status of all the Supercomputing
Centers, which I believe is a timely and worthwhile undertaking.

We have before us three relatively new reports, the most recent
of which, the Hayes Report, calls for some rather significant action
to be taken on this program.

I am hopeful that today's hearing will give us an opportunity to
understand why the Hayes Report calls for a departure from what
has been an unquestionably successful program.

Again, I am pleased that we are having this hearing. I hope it
will allow us to understand the significance of this program prior
to acting in any way to dismantle it.

Thank you very much, Mr. Chairman.

Mr. SCHIFF. Thank you, Mr. Doyle. I wanted to appreciate your
efforts on behalf of supercomputing both on behalf of the Pitts-
burgh Center and on behalf of the issue generally. It has been a
good contribution to this subcommittee.

Mr. Doyle. Thank you, Mr. Chairman.

Mr. SCHIFF. Now I want to say that all Members' statements,
without objection, will be made part of the record but if any other
Member desires to make an opening oral statement I will recognize
them at this time.

[No response.]

Mr. SCHIFF. I see no requests, so I am going to invite our first
panel.

Dr. Young and Dr. Hayes, please joint us at the witness table.

Gentlemen, while you are getting set up there, let me say that
your complete written statements will be made a permanent part
of this record and invite you to summarize or proceed as you think
best.

Dr. Young, please proceed.

STATEMENT OF DR. PAUL YOUNG, ASSISTANT DIRECTOR FOR
COMPUTER AND INFORMATION SCIENCE AND ENGINEER-
ING, NATIONAL SCIENCE FOUNDATION

Dr. Young. All right. I appreciate that, and I

Mr. SCHIFF. I think your microphone may not be on.

Dr. Young. Thank you. I appreciate that, and I will limit myself
to oral comments that I've made.

Chairman Schiff, Members, staff, guests:

I am Paul Young, Assistant Director of the National Science
Foundation's Computer and Information Science and Engineering.

I would like to begin by commenting how much I, and I am sure
all members of the Foundation and members of the science and en-
gineering communities generally, appreciate the comments of sup-
port and the efforts by Mr. Ehlers, by chairman Schiff, and by this
committee as a whole in support of both the 1996 budget, and basic
research and science and engineering generally.

My Directorate at the Foundation supports both research by, and
infrastructure, for the scientific and engineering communities. I am
here today to discuss our new program, Partnerships in Advanced
Computational Infrastructure, which is a follow-on to the very suc-
cessful NSF Supercomputer Centers Program begun in 1984.

Our research programs in computer science and engineering sup-
port basic research in computing information and communications.
Our organization's other infi-astructure program is in networking,
a program which has led to the current development of the
Internet, a subject about which you see daily references in the
press and elsewhere, and indeed Mr. Brown referenced the current
linking of the California schools, and we are very enthused about
that as well.

It is interesting to note that our two infi-astructure programs
began in the same NSF office about 1984 and have continued to
support and complement each other to this day.

I am going to first talk a bit about the background that led to
the establishment of the Supercomputer Centers Program, and
then briefly review some of its successes and the reasons for them.

I will then discuss the process that led to the restructured pro-
gram and its goals.

And finally, I would like to lay out the process and time line for
putting our new program in place and our commitment to a smooth
transition, a transition which will minimize the impact on the com-
munity of scientists and engineers who depend on us to support
their research.

When I finish, I would like to leave you with the following points:

First, the Supercomputer Centers Program was created in re-
sponse to a critical need of the research and education commu-

nities. It has not only succeeded in meeting that need, it has gone
well beyond.

Second, the science and engineering enabled by the program is
of extremely high quality. The need for continued support remains,
and is in fact increasing as computation becomes increasingly im-
portant in more and more research areas.

Third, advances in technology will now enable us to restructure
the program to involve a broader spectrum of individuals and insti-
tutions, chosen in a competition where the best ideas will be sought
and implemented.

Fourth, we at NSF are committed to a fair and open competition
using the best expertise in industry, academia, and government to
review proposals.

And finally, every effort will be made to maintain quality service
to the research community during the transition to the new pro-
gram.

The history of the Supercomputer Centers Program can be traced
back to the efforts of a number of people who realized that com-
putation could provide an important tool for research in science
and engineering, who also realized that advanced computation was
not generally available to the research communities that NSF tra-
ditionally supports.

In the early 1980s, NASA and DOE scientists had access to ad-
vanced computational tools for programmatic needs. NSF research-
ers whose work was related to these programmatic agency goals
could, in some cases get, access, but others could not.

For example, there were cases at that time where the only way
that NSF researchers could do their computational work was to go
overseas.

The realization of the need for computation to support a broad
range of research in science and engineering and the support of
many in Congress and in the Administration led to the establish-
ment of, first an access program; and then the NSF Supercomputer
Centers in the 1984-85 time range.

To say that this program has been a success is truly an under-
statement. Things that were not even imagined at the beginning
have been accomplished in the fields of science and engineering
that no one envisioned as having any need for computation have
been positively impacted.

It is often said that success has many parents and failure is an
orphan. If this can be quantified, then the degree of success must
be related to the number of people who have been a part of the en-
terprise over the last 11 years.

There are a multitude. Some of them are here. But since the in-
ception of the program, more than 20,000 people have used the pro-
gram in support of their research, and over 100 industrial firms
have used the program for training and to test the applicability of
supercomputing for their firms.

In one recent year alone, over 20,000 people had contact with the
Centers' education and training programs. Let me give a few exam-
ples of technology and research activities associated with the pro-
gram, but remind you that later speakers will continue to provide
instances of major successes in research and education.

One important far-reaching development in the success of the
Centers was the reahzation that researchers needed access from
their home institutions. Originally, several of the Centers even ran
their own networks; but as interest broadened, it became clear that
a national network was required.

The Centers were the original nodes on the NSFNet's National
Network, and the original nodes on the high-speed Net backbone,
and as such they played a major role in driving the networking
technologies needed for today's Internet.

Today they are playing a similar role in the evolution of NSF's
experimental, very high-speed network, the VBNS.

Another tool that assumed importance came from the realization
that the results of complex calculations could not be understood
from conventional computer output. The program has dramatically
enhanced the development of software tools to visualize the results
of calculations.

With the network connection, access to high-speed computers,
and visualization tools, computational scientists and engineers
were empowered to follow their imagination to new results and to
new understandings.

What are some of those research areas? And how is the field de-
veloping?

As one example, Gerry Ostriker, a cosmologist, who is now the
Provost at Princeton University, in briefing the National Science
Board and the Hayes Task Force, said that years ago he believed
that computers were having a negative impact on his area of re-
search. Promising graduate students would get intrigued with at-
tacking fi-ont-line physics problems with underpowered computers
and software, and then get absorbed in the technology.

Since then, the power of the machines and the understanding of
programming them have made enormous strides. Today, Ostriker
is one of the principal investigators in a team that has made im-
portant progress in modeling the formation of the universe.

The models are now so good that they can differentiate between
competing theories of the universe, and focus both theoretical and
observational work on the most fruitful lines of inquiry.

Another rapidly growing area of computation is in biology, where
physiologically interesting molecules can be successfully modelled
by computational methods and their behavior understood.

It is more and more likely that biologically active molecules — for
example, new drugs — can be designed and their behavior, both
good and bad, predicted with computational models.

Climate and meteorology research continue to benefit from com-
puting. Later this afternoon you will hear about storm prediction.

Work is also going on trying to understand the effects of green-
house gases on long-term climate change. These calculations re-
quire a sophisticated understanding of the interaction of the atmos-
phere and the ocean, which will have important other major bene-
fits to all of us.

Every time we think we understand which fields can benefit from
computational results, a new threshold in computing capability is
crossed and another field blossoms. As our progress has continued,
it can be truly said that computation has become a full partner in

the scientific method, along with experimentation, theory, and ob-
servation.

In 1993 a blue-ribbon panel chaired by Lou Branscomb issued a
report on The Future of Computation and the Tools Needed To Ex-
ploit It. Along with a truly ringing endorsement of the accomplish-
ments of the NSF's Centers Program, this report called for distrib-
uting the technology across a pyramid of capability from the apex
to work stations.

At the time of the Branscomb report, the budget of the national
High Performance Computing and Communications Program was
predicted to double in five years. While the Branscomb Report was
well received, budget realities prevailed.

The National Science Board, when confi-onted with a proposal to
continue the Centers without recompetition, was reluctant. Later
that year, the Board approved a two-year extension to the program
while a task force chaired by Ed Hayes evaluated a direction that
was more consistent both with budget predictions and with a rap-
idly evolving technology.

The Task Force on The Future of the NSF Supercomputers Pro-
gram met and conferred during calendar 1995, issuing a final re-
port to the NSF Director in September. The Task Force was an
independent group with many sectors represented, and it dealt
with the really hard issues.

The next speaker, Ed Hayes, was chairman of that Task Force
and will speak to its conclusions and the report. Based on the
Hayes Report and on deliberations within the Foundation, NSF
management decided in late 1995 to put forth a new program to
the National Science Board which was approved at their December
1995 meeting.

It is designed to officially take advantage of new partnerships,
explicitly including regional and state partnerships, and these in
turn take advantage of and drive both scalable parallel computing
and new advanced networking capabilities. The program solicita-
tion for the new program was released on the Worldwide Web just
prior — literally, the day of — the government shutdown on Decem-
ber 15th, 1995.

Later, a formal printed solicitation and a "Dear Colleague" letter
were issued, and a series of informational meetings were held on
both Coasts and in the Midwest.

As evidence of the accessibility of this information, and in spite
of the shutdown, our records show that over 1000 accesses were
made to the Web information before the shutdown ended, and that
over 3000 accesses have been made to the Hayes report and the so-
licitation from over 2000 distinct Internet addresses.

The current Centers Program and the new Partnerships Program
are designed to support the academic research community. As a
general rule, proposals and principal investigators at NSF are
drawn from academia. This is pointed out in the solicitation.

However, NSF is open to good ideas from any sector, and the cur-
rent program involves industry successfully in a variety of ways.
We will be pleased to see such involvement continue in the future,
and we will carefully review any new proposal that meets NSF's
broad acceptance criteria.

The time line and the process for a fair evaluation of proposals
is painstakingly detailed in our written testimony, and the solicita-
tion and the executive summary of the Hayes report are also in-
cluded.

In my opinion, the NSF has listened carefully to its constituents,
has paid close attention to the scientific needs of the Nation, and
is going forward with the program designed to capitalize on tech-
nical and budget realities.

We feel very confident that the new proposals we receive will be
very innovative, and they will be carefully reviewed using NSF's
best traditions of peer review.

We expect to go forward with a program that the Nation can be
proud of, and one that will continue the unparalleled successes of
the past.

Thank you, Mr. Chairman.

[The prepared statement of Dr. Young follows:]

Testimony for

Paul R. Young, Assistant Director

Computer and Information Science and Engineering

National Science Foundation

before the

Basic Science Subcommittee

House Committee on Science

March 19, 1996

Mr. Chairman and Members of the Subcommittee, thank you for the
opportunity to testify here today on NSF's new program of Partnerships for
Advanced Computational Infrastructure (PACI).

I am Paul Young, Assistant Director for NSF's Directorate for Computer and
Information Science and Engineering (CISE).

The new PACI program, which replaces the successful NSF Supercomputer
Centers program, is one important element in the Foundation's
implementation of the National High Performance Computing and
Communications (HPCC) Program.

At NSF the HPCC Program focuses on advancing the full range of advanced
computing, communications, information technologies, and infrastructure,
and supports fundamental research in these areas of computer science and
engineering, as well as the great diversity of applications in all areas of science
and engineering research which require HPCC. Thus, the research in
computer science and engineering supported in CISE is the prerequisite to the
development of future high performance computing, communications and
information systems, and forms the basis of an advanced National
Information Infrastructure. CISE also provides computing and
communications infrastructure for all research and education supported by
the Foundation through support for the Internet and the Supercomputer
Centers.

In this context, we expect that the new PACI Program will benefit from, and
provide new opportunities in, future high performance computing and
communications activities throughout the nation.

NSF's Early Participation in High Performance Computing

The National Science Foundation is now a major partner in providing the
nation's high performance computing infrastructure, but this was not always
the case. In the early 1970s the NSF ceased its support of campus computing
centers, and by the mid-1970s there were no "supercomputers" generally

a\-ailable to the academic community on any campus. Although computers of
this capability were available through other government agency (DoE and
NASA) laboratories, NSF did not play a role. As a consequence, most
academic researchers did not have the ability to perform computational
research on anything other than a departmental minicomputer, thereby
limiting the scope of research in many fields of science and engineering.

This lack of access to high performance computing was noted in the early
1980's in of a growing number of reports. For example a report to the NSF
Division of Phvsics Advisory Committee in March 1981 entitled "Prospectus
for Computational Physics", edited by William Press, identified a "crisis" in
computational physics, and recommended support for facilities. Subsequent
to this report a joint agency study, "Large Scale Computing in Science and
Engineering", edited by Peter Lax, appeared in December 1982 and acted as the
catalyst for NSF's reemergence in the support of high performance
computing. The Lax Report presented four recommendations for a
government-wide program:

• Increased access to regularly upgraded supercomputing facilities via high
bandwidth networks

• Increased research in computational mathematics, software, and
algorithms

• Training of personnel in scientific computing

• R&D for new supercomputer systems

The key suggestions contained in the Lax Report were studied by an internal
NSF working group, and the findings were issued in July 1983 as "A National
Computing Environment for Academic Research", a report edited by M.
Bardon and K. Curtis. The report studied NSF supported scientists' needs for
academic computing, and validated the conclusions of the Lax Report for the
NSF supported research community. The findings of Bardon/Curtis
reformulated the four recommendations of the Lax Report into a six point
implementation plan for the NSF. Part of this action plan was a
recommendation to establish ten academic supercomputer centers.

The immediate NSF response was to set up a means for academic researchers
to have access, at existing sites, to the most powerful computers of the day.
This was an interim step prior to a solicitation for the formation of academic
supercomputer centers directly supported by the NSF. By 1987, five NSF
Supercomputer Centers had been established, and all had completed at least
one year of operation.

During this early phase the Centers were essentially isolated "islands of
supercomputing" whose role was to provide supercomputer access to the
academic community. This aspect of the Centers' activities has changed
considerably. The NSF concept of the Centers' activities was mandated to be
much broader, as indicated by the Center's original objectives including:

• Access to state of the art supercomputers

• Nurture computational science and engmeermg m all fields

• Traming of computational scientists and engineers

• Encourage collaboration among researchers in academia, industry and
government

In 1988-1989 NSF conducted a review to determine whether support was
justified bevond 1990. In developing proposals, the NSF Centers were advised
to increase their scope of responsibilities. Quoting from the solicitation:

"To insure the long term health and value of a supercomputer center, an
intellectual environment, as well as first class service, is necessary. Centers
should identify an intellectual component and research agenda".

In 1989 NSF approved continuation through 1995 of the Cornell Theory
Center, the National Center for Supercomputing Applications, the Pittsburgh
Supercomputing Center, and the San Diego Supercomputer Center. Support
for the John von Neumann Center located at Princeton University vvas not
continued.

Scientific accomplishments of the Program

The Centers have fostered fundamental advances in our understanding of
science and engineering, expanded the use of high-end computing in new
disciplines, enabled the major paradigm shift to the acceptance of
computational science as a full partner in the scientific method, and
facilitated the education of a new generation of computational scientists and
engineers in support of that shift.

Several common themes emerge from examples of the impact of
supercomputing.

• First, the rapid growth of supercomputing together with its availability to
the research community have enabled computational science and
engineering to contribute to significant advances in a very wide and still
growing set of scientific and engineering fields.

• Second, high performance computing is making it possible to perform
complex simulations in three dimensions, rather than just two. This
important shift has dramatically enhanced the usefulness of
computational approaches, and will continue to do so at least through the
coming decade.

• Third, supercomputer-based simulations have combined multiple
disciplines and different physical phenomena to yield new scientific
discoveries and understanding.

• Last, increases in supercomputing capability and advances in
computational techniques are beginning to enable computer-based

bimulations to predict new scientific advances and to make new
discoveries.

In later testimony at this hearing the current Center Directors will highlight
the scientific and engineering accomplishments of the program in the last 11
vears.

Relevant Reports

Report of the Blue Ribbon Panel

Following the renewal of four of the Centers in 1990, the National Science
Board (NSB) asked the director of NSF to appoint a blue ribbon panel "... to
investigate the future changes in the overall scientific environment due [to]
the rapid advances occurring in the field of computers and scientific
computing." The resulting report, "From Desktop to Teraflop: Exploiting the
U.S. Lead in High Performance Computing," was edited by Lewis Branscomb,
and was presented to the NSB in October, 1993.

This report pointed to the Foundation's accomplishments in the seven years
since the initial implementation of the recommendations of the Lav Rr^port
on high performance computing (HPC) and the establishment of the
Supercomputer Centers. The report asserted that the NSF Centers had created
an enthusiastic and demanding set of sophisticated users who make
fundamental advancements in their scientific and engineering disciplines
through the application of rapidly evolving high performance computing
technology. Other measures of success cited include the thousands of
researchers and engineers who have gained experience in HPC, and the
extraordinary technical progress in realizing new computing environments.

The report noted that, through the NSF program and those of sister agencies,
the U. S. enjoys a substantial lead in computational science and in the
emergmg, enabling technologies. It called for NSF to capitalize on this lead,
which not only offers scientific preeminence, but also aids the associated
industrial lead in many growing world markets.

Primary recommendations included the following:

• The NSF should retain the Centers and reaffirm their mission with an
understanding that they now participate in a much richer
computational infrastructure than existed at their formation.

• The NSF should assist the university community in acquiring mid-
range systems to support scientific and engineering computation and to
break down the software barriers associated with massively parallel
systems.

• The NSF should initiate an interagency plan to provide a balanced
teraflop system, with appropriate software and computational tools, at
the apex of the computational pyramid.

These recommendations and the accompanying challenges could be
summarized as calling for a broad based infrastructure and research program
that would not only support the range of computational needs required by the
existing user base, but would also broaden that base in terms of the range of
capabilities, expertise, and disciplines supported.

Report of the NRC-HPCC Committee

In 1994, Congress asked the National Research Council (NRC) to examine the
status of the High Performance Computing and Communications Initiative.
The NRC committee stated that it "believes that strong public support for a
broadly based research program in information technology is vital to
maintaining U.S. leadership in information technology". While the
committee did not make explicit recommendations for funding levels or
management structures for the Supercomputer Centers program, it did say:

"The committee recognizes that advanced computation is an
important tool for scientists and engineers and that support for
adequate computer access must be a part of the NSF research program
in all disciplines. The committee also sees value in providing large-
scale, centralized computing, storage, and visualization resources that
can provide unique capabilities. How such resources should be funded
and what the long term role of the Centers should be with respect to
both new and maturing computing architectures are critical questions
that NSF should reexamine in detail, perhaps via the newly
announced Ad Hoc Task Force on the Future of the NSF
Supercomputer Centers Program."

The Task Force on the Future of the NSF Supercomputer Centers Program

The Task Force met during NSF's 1995 Fiscal Year to advise NSF on several
important issues related to the review and management of the NSF
Supercomputer Centers program. The Task Force was charged to analyze
various alternatives for the continuation, restructuring, or phase-out of
NSF's current Supercomputer Centers program, or the development of
similar future program(s), and to make recommendations among the
alternatives.

The final report of the Task Force was presented to the NSF Director on
September 15, 1995. The following language is excerpted from the executive
summary of the report, which is appended to this testimony.

The Task Force believes that the future for computational science and
engineering can be as bright or even brighter than in the past decade. If we
seize the opportunity, over the next decade we can make major progress on

multiple fronts.

There will be

• opportunities for exciting applications of our nation's exponentially
increasing computational capacity, for example:

- more complete models, and hence deeper understanding of
physical systems by moving to three and higher dimensions;

- progress in computational tools to aid drug and protein design;

- computational predictions of scientifically and commercially
significant materials;

- multidisciplinary models of physical systems (e.g., combining
fluid dynamics and electromagnetic models of the heart);

- increased interconnectivity of supercomputers and high impact
instrumentation; and

- models of anatomical and physiological processes leading to nezv
insights of benefit to human health.

• more quantitative computational results in unanticipated areas.

• more explosive groxvth of communications as a component of the
computational science and engineering paradigm; and, importantly,

• continued progress in the tools and methods for developing code that
is both portable and yet takes advantage of unique parallel
architectures.

These advances will not automatically become available to American
researchers. To position the U.S. academic community to participate in the
exciting research possibilities enabled by these developments, the Task Force
has the follozving recommendations leading to a restructured Centers
program.

In order to maintain world leadership in computational science and
engineering, NSF should continue to maintain a strong, viable Advanced
Scientific Computing Centers program, whose mission is:

• providing access to high-end computing infrastructure for the
academic scientific and engineering community;

• partnering with universities, states, and industry to facilitate and
enhance that access;

• supporting the effective use of such infrastructure through training,
consulting, and related support services;

• being a vigorous early user of experimental and emerging high
performance technologies that offer high potential for advancing
computational science and engineering;

• facilitating the development of the intellectual capital required to
maintain world leadership.

NSF should abSiire that the Centers program provides national "Leading-edge
>ites" that have a balanced set of high-end hardware capabilities, coupled with
appropriate staff and software, needed for continued rapid advancement in
computational science and engineering.

NSF, through its Centers program, should assure that each leading-edge site is
partnered with experimental facilities at universities, NSF research centers,
and/or national and regional high performance computing centers.
Appropriate funding should be provided for the partnership sites.

NSF should announce a new competition of the High Performance
Computing Centers program that would permit funding of selected sites for a
period of five years. If regular reviews of the Program and the selected sites
are favorable, it should be possible to extend initial awards for an additional
five years without a full competition.

The Centers program should continue to support need-based research in
support of the program's mission, but should not provide direct support for
independent research.

NSF should increase the involvement of NSF's directorates in the process of
allocating service units at the Centers.

NSF should provide leadership in working toward the development of
interagency plans for deploying balanced systems at the apex of the
computational pyramid and ensuring access to these systems for academic
researchers.

National Science Board Actions

In October of 1994, the National Science Board (NSB) approved a two year
"extension of the original Centers program (through FY 1997) in order for the
Task force to complete its work and to allow time for a new competition, if
NSF recommended one and the NSB agreed.

The Task Force report was presented to the NSF Director on September 15,
1995 and to the NSB at the October 1995 meeting on the Committee on Plans
and Policy. The NSF recommendations to the NSB were scheduled for
presentation at the November meeting, but were postponed to December by
the first government shutdown. At their December meeting the NSB
approved a new competition for a program called Partnerships in Advanced
Computational Infrastructure (PACI).

The Program solicitation for the Partnerships meeting was released on the
World Wide Web (WWW) on December 15, 1995, immediately prior to the
second FY96 shutdown. Since that time a printed solicitation, a dear colleague
letter and a series of informational meetings have been held to insure that
the community was aware of the program and had all the required
information to propose to participate. In addition, a set of frequently asked
questions and their answers has been posted on the WWW and kept

continuously updated. The complete program solicitation is appended to this
testimony.

Partnerships program competition

Mission

As part of its strategic plan, NSF in a Changing World, a key NSF goal is to
enable the United States to uphold a position of world leadership in science
and engineering research and education. In order to maintain world
leadership in computational science and engineering, NSF intends to create
an adyanced national computational infrastructure whose overall mission is
to:

• provide, facilitate, and enhance access to high performance computational
infrastructure for the U. S. academic, scientific, and engineering
communities by partnering with universities, states, and the private
sector;

• promote vigorous early use of experimental and emerging high
performance computational and associated communications technologies
that offer high potential for advancing science and engineering;

• enable the effective use of such infrastructure and technologies through
education, training, consulting, and related support services, including
appropriate software development, experimentation, and support;

• foster interdisciplinary research in science and engineering;

• facilitate the development of the intellectual capital required to maintain
world leadership in computational science and engineering; and

• broaden the base for the nation's advanced computational and
communications infrastructure.

Solicitation

The program solicitation for the Partnerships for Advanced Computational
Infrastructure program, builds on and replaces the current NSF
Supercomputer Centers program, and focuses on taking advantage of newly
emerging opportunities in high performance computing and
communications. The new program will provide flexibility, both to adapt to
rapidly evolving circumstances and to meet the need for high-end
computation, in order to enable continued world leadership in
computational science and engineering. The program will provide access for
researchers to high performance computing systems as its core, with
associated highly trained staff and researchers necessary to develop and
optimize their use. The emergence of scalable parallel systems, high
performance networking and high bandwidth, large capacity mass storage
systems creates the opportunity for a national infrastructure consisting of a
number of geographically distributed sites strongly coupled to high-end

computational resources and to each other \'ia high-speeci communication
networks.

NSF envisions an Advanced Computational Infrastructure consisting of one
or more leading-edge sites together with cooperating partners. Leading-edge
sites are expected to maintain high-end hardware systems that are one to two
orders of magnitude more capable than those typically available at a major
research university. These systems should be balanced in terms of processor
^peed, memorv, and storage systems, and should be accompanied by
appropriate staff, software and high speed communications capability. The
partners will, in the aggregate, complete the overall infrastructure by, among
other things, (a) facilitating research and experimentation with new hardware
and software, including appropriate support technologies such as
visualization and mass storage, (b) providing scalable resources for
applications and applications development that can be best done on mid-level
systems, (c) providing access to unique experimental systems and facilities,
and (d) promoting education and training.

Smce the Partnerships for Advanced Computational Infrastructure Program
will primarily support academic research, it is expected that proposals will
come from, and the partnership's cooperative agreement will be with, a U. S.
academic institution. However, a successful proposal must involve multiple
partners who can be, but are not limited to:

• universities, including research groups within universities;

• NSF-funded Centers and facilities, such as. Science and Technology
Centers, Supercomputer Centers, Engineering Research Centers, and
Industry-University Cooperative Research Centers;

• research and educational consortia, organizations, and groups;

• regional and state-supported high-performance computing centers;

• private sector organizations; and

• federal laboratories.

A full description of organizations and individuals who are eligible to submit
proposals to NSF and the conditions under which they may compete, are

given in the NSF Grant Proposal Guide (GPG) (NSF 95-27).

Review Process

Overview

The review process will involve three stages; preproposals, full proposals and
site visits. The preproposals, due April 15, 1996, will be evaluated to
determine who will be encouraged to submit a full proposal. The intent of
the preproposal round is to give the community an opportunity to put forth
ideas, to provide proposers with information that will assist in developing a

stronger proposal, and to identity at an early stage, preproposals that are not
expected to be competitive. Feedback will be provided to all preproposal
submitters; they may submit a full proposal regardless of the opinions
expressed by the reviewers.

Full proposals are due September 1, 1996. Both preproposals and full
proposals will be evaluated by external panels. In late fall, 1996,
recommended proposals will be subjected to a site visit (but not necessarily all
sites in a given partnership) in order to clarify issues raised during the panel
review, and to explore additional matters as needed. A summary panel,
made up of the chairs of the site visit teams and the chair of the full proposal
review panel, will meet after the site visits.

Criteria

As with all proposals to NSF, these proposals will be subject to the four
standard review criteria, i.e. the quality of the proposed scientific effort,
competence of the investigators, relevance of the research, and impact on the
infrastructure of science and engineering. However, for this particular
solicitation, impact on infrastructure and competence of investigators will be
especially important. Thus both preproposals and final proposals for the
Partnerships for Advanced Computational Infrastructure program will be
subject to four, more specific, criteria dealing with, the effectiveness of the
overall partnership in addressing the program's mission, the quality of the
individual partners, the effectiveness of the proposed management, and the
degree of financial leverage. While these four, more specific, criteria just
mentioned are listed in priority order (with the first being the most
important), all will be considered by the panels in arriving at
recommendations. Finally, the summary panel will be asked to advise NSF
on what each proposed partnership can contribute to the technical diversity
and balance of the program as a whole.

Management

The review process will be managed by the Division of Advanced Scientific
Computing (ASC). In order to assure that all NSF directorates are involved
in the process, an NSF internal PACI Review Committee will be established
chaired by the Centers Program Director in ASC. Each directorate will be
invited to appoint one or more representatives to the committee. Ideally,
there should be strong overlap between the membership of the High
Performance Computing and Communications Coordinating Committee and
the Review Committee. The Review Committee will have the following
responsibilities:

• Nominate panelists and site visitors for the review processes outlined
below.

• Participate in review panels, including serving as moderators of possible
hubpanels.

• Represent their directorate on site visits to candidate partnerships.

• Pro\ide a knowledgeable link between their directorate and the process.

Structure of Proposal Review Panels

Panelists will be chosen from a variety of sectors: academia, the private sector,
national laboratories, and other government agencies. Since the academic
community will be the major focus of the PACI Program, subject to possible
conflict of interests, a majority of reviewers will be chosen from this sector.
As described above, recommendations from all NSF directorates for possible
panelists will be solicited through the PACI Review Committee. Reviewers
will be chosen from the computational science and engineering and
computer science and engineering communities whose areas of expertise
represent those of users of the infrastructure, enablers of the underlying
technologies, and managers of large facilities, and encompass both research
and educational needs for the PACI program.

Preproposal panel procedure

Written reviews of each preproposal will be provided by at least 3 readers, and
a summary of the subpanel discussion of each preproposal will be recorded on
a panel review form which highlights the major review criteria. The task of
the subpanels will be to recommend whether a full proposal is to be
encouraged or not. A target number of proposals to be encouraged will be
suggested to the panels. All reviews and the panel summary will be provided
to the PI.

No PI or named collaborator will be chosen to be a reviewer (panelist) due to
disqualifying conflict of interest. Reviewers from the same institution may
have to be used due to the large number of partners expected in the
submissions, but an attempt will be made to segregate these into subpanels
considering preproposals not involving their institution; if this is not
possible, the reviewer will be absent from the discussion of any preproposal
involving his/her institution.

Full proposal panel procedure

The panelists will be chosen from the original subpanels reviewing the
preproposals, from the pool of submitters of preproposals not submitting full
proposals, and from the community at large. While there will be some
overlap between the panels reviewing the preproposals and the final
proposals, the panelists will be instructed that any recommendations based on
the preproposal review were not intended to be binding on the proposer and
should not be considered in reviewing the final proposal. Preproposal
reviews and recommendations will not be available to the panelists.

Each reviewer will be provided with all of the proposals and a specially
prepared review form which will reflect the major review criteria. Each
proposal will have at least 3 written reviews prepared by reviewers prior to
the panel discussion; each panelist will be assigned approximately 4 of the

proposals tor which he/she must prepare written reviews. These written
reviews will include ratings categorizing the assigned proposals into the top
25"/u, the next IS"";., and the remainder. All proposals will then be discussed in
the panel. The results of the panel discussions will be recommendations
which rate the proposals as "definitely site visit", "possibly site visit", and "do
not consider further". All reviews and a panel summary will be provided to
the PI.

It is expected that conflicts will be less than in the preproposal stage, and any
that arise will be handled in a similar manner.

Site visit procedure

NSF staff will review the recommendations for site visits. In order that the
portfolio for site visits assures consideration of a balanced program, NSF may
possibly augment the "definitely visit" group with some from the "possibly
visit" group.

All proposals recommended will be reviewed by a team of site visitors
consisting of previous panelists augmented by other reviewers as appropriate.
Partnerships to be visited will be provided with a set of questions prior to the
visit. NSF staff members, including members of the PACI Review
Committee described above, will be invited to participate in all site visits as
observers. Each site visit team will have a chair designated by NSF. Each
visitor will go on at least two site visits (preferably more) in order to have an
overview of the possible varieties of partnerships available. The
recommendations of the team will be to either fund, fund if possible, or not
fund the partnership.

Upon completion of all site visits, a panel of the site visit chairs, chaired by
the chair of the proposal review panel, will meet at NSF to review the overall
recommendations made by the site visit teams, and to provide NSF with
several options (e.g., one large partnership, or several smaller partnerships)
which meet the overall mission of the PACI Program.

Management

Active management of the program by the Foundation is essential, since the
new program has been designed to provide a national infrastructure that is
built on one or more multi-institutional partnerships. Because of the
complexity and distributed nature of the new program, both external and
internal management issues must be considered. The management model
draws on experience gained with a number of other multi-institution NSF
programs, but management of this program is still expected to offer unique
challenges.

Each partnership will work with NSF to develop criteria by which it can be
evaluated. These criteria will be used in all formal reviews. Each partnership
will submit an annual program plan which summarizes its progress and
describes activities and budgets for the coming year. This may include

proposed changes in membership, research, service, networking, computing
bvstems, or funding changes for its members. These plans will be
indi\-iduaily reviewed by an external program review panel appointed by
NSF which will provide recommendations to NSF regarding funding levels,
addition of or deletion of partners, new opportunities, etc. Although they
will not be members of the panel, it is expected that representatives of all
partners in each partnership will be at these reviews and that their locations
may vary between partnership sites and NSF.

To provide flexibility in meeting special needs, funding can be provided for
partnerships having a membership that may change over the duration of the
ceioperative agreement; thus initial proposals involving partners for a shorter
period are encouraged, where appropriate. Furthermore the partnership can
propose, at any time, a change in the number or composition of its partner
sites. Normally such proposals will be part of the annual program plan
review process and would be considered along with other substantive issues.
Possible justifications for such a change might be the occurrence of an
unusual opportunity, completion of a partner's task, changes or relocation of
a critical person, etc.

More frequent informal monitoring will be carried out by NSF staff. Each
partnership will be required to contribute to a database relating to usage,
users, training, outreach, education, industrial interactions, and other
significant data relating to the operation of the program. NSF will also
require periodic meetings of the Pi's and co Pi's of the partnerships for overall
program coordination. This will permit the Foundation to better understand
and monitor both the operation of a partner within the partnership, the
partnership itself, and the program as a whole.

A single institution, through the PI, is expected to be responsible for overall
management and leadership of the partnership as a whole, and each partner
site will have a local director. Partnerships will have a policy and operations
management committee that will meet regularly. Its membership will
consist of a senior representative from each site. NSF will host an annual
meeting of representatives from all the partnerships to discuss program wide
coordination.

Every partnership will be encouraged to appoint an independent external
advisory committee which will provide oversight and guidance necessary for
the management of the partnership. This committee will include users of the
partnership resources and others who are in a position to evaluate the
activity and provide feedback to its management. This group will be selected
in consultation with program management staff and will meet at least once a
year.

The Transition

The transition from the old Centers Program to the new Partnerships
Program will take place beginning immediately after Board approval of the
new partnerships.

It is critical that this transition be conducted smoothly to ensure that there is
continuity in the availability of high performance computing resources to
support the research of the academic science and engineering community.

The final transition plan will be designed in accordance with the principles
below:

• minimum disruption for the scientific research community;

• deferral of all FY97 Supercomputer Centers program expenditures
possible, until awards for the new program are announced;

• rapid phase out of old participants; rapid phase in of new partners;

• immediate and active NSF involvement in the transition process.

Summary

At the conclusion of this exercise, NSF will continue world class support of
computational science and engineering for at least another 5 years. During the
fourth year of the program an evaluation will determine whether to continue
without recompetition for another five years at which the program will be
"sunset". Before this occurs another review will determine the future for
continued NSF support of the infrastructure.

Mr. SCHIFF. Well, Dr. Hayes, you got quite an introduction from
the first witness, so with that I will call upon you, please.

STATEMENT OF DR. EDWARD HAYES, CHAIRMAN, REPORT ON
THE TASK FORCE ON THE FUTURE OF NSF
SUPERCOMPUTING CENTERS PROGRAM AND VICE PRESI-
DENT FOR RESEARCH, OHIO STATE UNIVERSITY, COLUM-
BUS, OHIO

Dr. Hayes. Thank you, Mr. Chairman.

I would also like to add my thanks to you and to the Committee
for their strong interest in the Supercomputing Centers Program
and their bipartisan support over the years for the science pro-
grams funded by the Federal Government.

In addition to being the chair of the Task Force, my da5rtime job
is vice president for research at the Ohio State University; and my
research interests are in the area of computational chemistry.

I am very pleased to have this opportunity to talk with you today
about the new partnerships for Advanced Computational Infra-
structure, a Program that was recently announced and that Paul
has been talking about.

My objective is to give you a context as well as a brief overview
of the Task Force Report. A key message that I would like to leave
with the Committee today is how pleased the Task Force is with
the process that NSF has followed in setting the directions for the
future of its highly successful Supercomputing Centers Program.

The members of the Task Force are listed in my prepared re-
marks. This is a group with exceptional talent and breadth of expe-
rience. It made the job of the chair quite enjoyable, as well as chal-
lenging.

All are knowledgeable about one or more aspects of high-perform-
ance computing. They are all independent-minded and committed
to doing their work in a professional way. They are pleased, and
we were pleased to be able to report the unanimous view of what
steps should be recommended to the National Science Foundation
on the future of the program.

In essence, we found that the Supercomputing Centers Program
has been a critical component of the strong U.S. position in science
and technology, and we strongly recommended that it be continued
but with important changes.

With the issuance of the new Partnerships Program announce-
ment, the National Science Foundation initiated the important pro-
posal development stage. At this stage, various groups of univer-
sities and other potential partners are putting their best ideas to-
gether to see if they will be able to convince their peers that they
have the best plans to enable the future of computational science
and engineering.

While our Task Force set the stage for the final program an-
nouncement, and the program announcement set the stage for
these proposal development efforts, there is still significant room
for creativity on the part of the Partnership Centers.

The Task Force clearly understood the importance of not tying
the hands of creative people at the proposal development stage.
From this perspective, we are pleased with the clarity of focus ex-

emplified in the program announcement, as well as the appro-
priateness of the selection criteria.

The competitive responses often bring out important new ideas,
insights, and commitments that cannot be accurately forecast in
advance. The final stage of the process will begin after the propos-
als are received and the merit review begins.

The Task Force has a very high level of confidence in Paul
Young, the assistant director, who has line management respon-
sibility for this Program. He and his staff are most knowledgeable
in this area and have a great track record for running fair and ef-
fective review processes.

At this point I want to make just a few brief comments from my
statement that focus on just a couple of aspects of the program.

The Task Force heard from many knowledgeable individuals
from industry, academia, government, and government labora-
tories, NSF assistant directors, academic researchers, representa-
tives of other federal agencies, center directors, as well as members
of the centers allocation committees. We talked with them all.

Considerable input was received from many thoughtful people.
This input, as well as our survey to which there were 500 replies,
was essential. It extended the knowledge base of the individual
members of the Task Force, and informed us about the key issues
surrounding the Centers Program.

This input also gave us an excellent sense of the important role
that the Centers Program has played in making computational
science and engineering a significant contributor to the long-term
progress of fundamental research and, importantly, in providing
the infrastructure needed for education and training of future gen-
erations of world leaders in science and technology.

The recommendations of the Task Force follow from our vision of
the future of high-performance computing. This vision has several
key points. I want to mention just one of these — there are seven
in total — this afternoon. The others are listed in my prepared re-
marks.

A new element of our vision is a strong coupling of selected re-
search centers and university laboratories with the leading edge
sites that have the highest end computational systems.

We believe that such coupling has great potential for the future
in terms of an enhanced program flexibility, creativity, and effi-
ciency.

With advances in high-speed communications, this approach will
also provide the infrastructure needed to support creative experi-
ments with distributed computing.

In my prepared remarks I have commented on each of the Task
Force recommendations. Two of these recommendations are central
to the new Partnerships Program.

In our second recommendation we noted that NSF needs to sup-
port a few leading edge sites, sites that have a balanced set of
high-end hardware capabilities, coupled with appropriate staff and
software. Raw processor speed or compute- power is not sufficient
to achieve the balanced high-performance systems needed for para-
digm-shifting research.

In our third recommendation we focused on partnerships. Part-
nerships will be important for the future of the program. With stra-

tegic coupling between leading-edge sites and research centers and
university laboratories, it is a significant new change.

These partnership sites will provide better coupling to computer
scientists developing new tools and software. This coupling will be
particularly important when their research on mid-range systems
reaches the point when it needs to examine scalability issues that
become manifest only on the leading edge systems. But the impact
of this research will extend beyond computer science.

The partnership sites will also provide more cost-effective and di-
verse platforms for early design and testing of applications soft-
ware prior to the stage where access to a leading-edge system is
required.

Although we began with the work of earlier reports, we did not
accept the conclusions or the recommendations uncritically. We car-
ried out our own analysis particularly of the merit review process
for allocating resources at the current centers, an analysis of the
educational benefits of the program, and we also sought significant
input from the community, including several key NSF advisory
committees particularly on the benefits of the current program.

In preparing for this hearing I was able to talk to most of the
members of the Task Force to confirm their enthusiasm for the
manner in which NSF management is moving forward with the
program announcement and the receipt of proposals. All indicated
that they are pleased with the progress that has been made up to
this point.

Thank you, Mr. Chairman.

[The prepared statement and attachments of Dr. Hayes follow:]

Statement of Dr. Edward F. Hayes

Vice President for Research

The Ohio State University

AND

Chair, Task Force on the NSF Supercomputer Program

Mr. Chairman and Members of the Subcommittee:

I am very pleased to have this opportunity to talk with you about the New Part-
nerships for Advanced Computational Infrastructure program that was recently an-
nounced by the National Science Foundation.

One of the relevant reports that guided the National Science Foundation in the
design of this new program was the Report of the Task Force on the Future of the
NSF Supercomputer Program. It was my honor to be asked by the Director of the
National Science Foundation to chair this Task Force.

My objective today is to give you a context as well as a brief overview of the Task
Force Report. A brief overview is not an easy task because our report deals with
many significant questions and issues. A key feature of our report is the extent to
which we dealt with the tough issues and seriously considered alternatives.

A key message that I would like leave with the committee is how pleased the
Task Force has been with the process that NSF has followed in setting the direc-
tions for the future of its highly successful Supercomputer Centers Program.

The Task Force was appointed by the Director of the Foundation, Dr. Neal Lane,
and asked to provide him with recommendations on the future of the Supercomputer
Centers Program and to report our findings as soon as possible. The work of the
Task Force began in January of 1995 and culminated in our final report to the Di-
rector last September.

The members of the Task Force are listed below:
• Dr. Arden L. Bement, Jr., Basil S. Turner Distinguished Professor Of Engineering,
Purdue University

• Chairman — Dr. Edward F. Hayes, Vice President for Research, The Ohio State

University

• Dr. John Hennessy, Chair Computer Science, Stanford University

• Dr. John Ingram, Schlumberger Research Fellow, Schlumberger, Austin

• Dr. Peter A. KoUman, Professor of Chemistry and Pharmaceutical Chemistry, Uni-

versity of California — San Francisco

• Dr. Mary K. Vernon, Professor of Computer Science, University of Wisconsin

• Dr. Andrew B. White, Jr., Director of Advanced Computing Laboratory, Los Ala-

mos National Laboratory

• Dr. William A. Wulf, AT&T Professor of Engineering and Applied Science, Univer-

sity of Virginia
Ex-officio Non-Voting members:

• Dr. Robert G. Voigt, NSF, Acting HPCC Coordinator

• Dr. Paul R Young, NSF, Assistant Director, CISE

• Dr. Nathaniel G. Pitts, NSF, Director/OSTI

There is considerable talent and experience in this group. Some of these names
may be famihar. All are knowledgeable about one or more aspects of high perform-
ance computing. They are all independent minded and committed to doing their
work in a professional way. We did not start out with a common view of what rec-
ommendations we might make to the National Science Foundation. In fact, we had
many discussions in which one or more members of the group were observed in var-
ious stages of testiness. However, I believe that we were all pleased to be able to
report that after a great deal of effort on everyone's part we did come to a unani-
mous view of what steps should be recommended. In essence, we found that the
Supercomputer Centers Program has been a critical component of the strong U.S.
position in science and technology and we strongly recommended that it be contin-
ued, but with important changes.

After receiving our report, the Director of the Foundation initiated the second
stage of the process. During this stage he reviewed the report with key staff within
the agency and formulated an action plan to be discussed with the National Science
Board for their review and approval. The National Science Board provides a very
important perspective on major Foundation programs because they represent a very
broad cross section of expertise and considerable experience with policy issues and,
importantly, experience in setting priorities in the context of the whole NSF and
Federal Research Budgets. At the October 1995 meeting of the National Science
Board, Arden Bement, another member of the Task Force, and I were asked to pro-
vide members of the Board with an overview of the Task Force Report. At this meet-
ing the Board also heard from Fred Brooks, co-chair of the NRC report on the HPCC
Program, and Jerry Ostriker, an eminent computational scientist who has made
many significant contributions to our understanding of cosmology.

Following the October meeting of the National Science Board, the third stage of
the review process began as the NSF Director put together the proposed program
announcement for the new partnerships program. The Board was scheduled to dis-
cuss and act upon this proposal at its November meeting, but strange things were
happening with the FY 1996 Budget discussions during this period — unfortunately
the NSF was shut down during the period of the scheduled November Board meet-
ing. Ultimately, the NSF was able to stay open long enough for the Board to meet
in December — at which time the program announcement was given final approval
and was quickly put out on the streets just before the government shut down again.

The issuance of the Program Announcement began the fourth stage in the proc-
ess — the important proposal development stage is when various groups of univer-
sities and other potential partners put their best ideas together to see if they will
be able to convince their peers that they have the best plans to enable the future
of computational science and engineering. While our Task Force set the stage for
the final program announcement and the program announcement set the stage for
these proposal development efforts, there is still significant room for creativity on
the part of the partnership centers. The Task Force clearly understood the impor-
tance of not tying the hands of creative people at the proposal development stage.
From this perspective, we are pleased with the clarity of focus exemplified in the
program announcement as well as the appropriateness of the selection criteria. The
competitive responses at this stage often bring out important new ideas, insights
and commitments that cannot be accurately forecast in advance.

The final stage will begin after proposals are received and the merit review be-
gins. The Task Force has a very high level of confidence in Paul Young, the Assist-
ant Director, who has line-management responsibility for this program. He and his
staff are most knowledgeable in this area and have a great track record in running

fair and effective review processes. If we can just keep the agency open, they will
do a great job in picking the winners.

At this point, I want to return to the report of the Task Force.

The current NSF Supercomputer Centers Program was initiated in the 1985-86
time period with the formation of five supercomputer centers. Each one had a vector
supercomputer that was considered big for the time period. Over the first five years
of the program's existence these centers provided vector supercomputing services for
the research community and training for the many researchers who lacked experi-
ence with such systems. Prior to 1985, U.S. academic researchers had very limited
access to vector supercomputers.

At the end of the first five years, NSF held a major external review of the five
Centers. Four of the five Centers were approved for continuation for another five
years. The John von Neuman Center in New Jersey was not renewed.

In the second five years of the Center Program's existence several significant
changes were made:

• There was a distinct effort to expand the outreach efforts of the centers and to

forge closer ties with industryy.

• The four remaining centers — at Cornell University, the University of Illinois, the

Pittsburgh Center and the San Diego Center — formed themselves into a
MetaCenter with resources sharable on a national scale and coordinated plan-
ning to develop and not duplicate areas of special expertise within the program.

• Finally there was a move to change the computational paradigm as a new genera-

tion of parallel architecture machines started hitting the market.

So it was after ten years and many successes that our Task Force was called to-
gether to advise the National Science Foundation on the future of the Centers Pro-
gram.

In arriving at our conclusions, we benefited significantly from two earlier reports
that I want to comment on briefly because of their direct relevance to our own work.

From Desktop Teraflop: Exploiting the U.S. Lead in High Performance Computing,
NSF Blue Ribbon Panel on High Performance Computing, August 1993

Evolving the High Performance Computing and Communications Initiative to Sup-
port the Nation's Information Infrastructure, Computer Science and Telecommuni-
cations Board, National Research Council, 1995

Both of these reports speak to the importance of exploiting the U.S. lead in high
performance computing.

The Blue Ribbon Panel emphasized the importance of maintaining a balance in
the pyramid of computational capability. See Attachment 1.

The NRC/HPCC Committee made several important observations central to the
work of our Task Force.

I would like to quote a few sentences from the executive summary of the HPCC
report. These quotes capture for me, in a succinct way, the importance of high per-
formance computing for the nation.

In the section dealing with high performance we read:

" Tiigh performance' — which involves bringing more powerful computing and com-
munications technology to bear on a problem — has enabled advances on several
fronts. High performance system.s, for example, deliver answers sooner to complex
problems that need large amounts of computing. Timely and accurate forecasting of
weather, mapping of oil reservoirs, and imaging of tumors are among the benefits
encompassed by the goals of the HPCC initiative."

Then continuing on in the next paragraphh.

"Information technology evolves as new and valuable applications are found for
hardware that gets steadily more powerful and cheaper. To benefit, users need af-
fordable hardware, but they also need the software that implements the new appli-
cations. Yet learning how to build software takes many years of experimentation.
If this process starts only when the hardware has already become cheap, the bene-
fits to users will be delayed by years. Research needs to treat todays expensive
equipment as a time machine, learning how it will be used when it is cheap and
widely available, as surely it will be tomorrow."

In addition to the insights obtained from these two reports, the Task Force heard
from many knowledgeable individuals from industry, academia, government, and
government laboratories. NSF Assistant Directors, Academic Researchers, Rep-
resentatives of other Federal Agencies, Center Directors as well as members of the
centers allocation committees — we talked with them all. Considerable input was re-
ceived from many thoughtful people. This input as well as our survey, to which over
500 replied, was essential. It extended the knowledge base of the individual mem-
bers of the Task Force and informed us about the key issues surrounding the Cen-
ters Program. This input also gave us an excellent sense of the important role that

the Centers program has played in making computational science and engineering
a significant contributor to the long term progress of fundamental research, and, im-
portantly, in providing the infrastructure needed for the education and training of
future generations of world leaders in science and technology.

Today, much of the important computational science and engineering is being car-
ried out on workstations and mid-range machines. That is desirable. It is one of the
clear signs of the health of the U.S. high performance computing program. A signifi-
cant amount of the applications software that is being run on today's workstations
was developed during the early days of the NSF Supercomputer Centers Program.
The systems of that period had about the same power as today's workstations.
These earlier "time machines" worked just the way people predicted that they would
ten years ago.

The recommendations of the Task Force follow from our vision for the future of
high performance computing. This vision has seven key points that I summarize
below.
Future Vision

• Computational Science & Engineering is central to the long term development of

fundamental principles and understanding of complex systems

• Continued U.S. leadership in computational science and engineering requires ac-

cess to the highest end computational systems, systems that are balanced with
respect to processor speed, memory, storage and software

• Coupling of selected mid-range systems to the leading-edge systems has great po-

tential in the future in terms of enhanced program flexibility, creativity, and
efficiency

• Research on computing tools, algorithms and new models of physical systems pays

significant dividends, but such research takes time and commitment particu-
larly with changes in the underlying computing paradigm

• Unanticipated results are often the most important

• Research trains people for the future needs of the nation

• SjTiergy among industry, academia, and government

Computational science and engineering is central to the long term development
of fundamental principles and the understanding of complex systems. Computa-
tional science brings enhanced synthesis and analysis. It is the source of new in-
sights that facilitate a deeper understanding of fundamental issues and, impor-
tantly, computational science is a full partner with experimental and theoretical
science in extending the analysis of experimental results that cannot be fully syn-
thesized without the assistance offered by sophisticated computational models.

Continued leadership in computational science and engineering requires access to
the highest end computational systems. Here we mean systems that are balanced
with respect to processor speed, numbers of processors, memory, storage, software
and personnel. These systems act as time machines for academic researchers allow-
ing them to gain significant lead time in addressing computational problems and is-
sues that emerge whenever significant new levels of modeling capability emerge.

A new element in our vision is a strong coupling of selected research centers and
university laboratories with the leading-edge sites that have the highest end com-
putational systems. We believe that such coupling has great potential for the future
in terms of enhanced program flexibility, creativity, and efficiency. With advances
in high speed communications this approach will also provide the infrastructure
needed to support creative experiments with distributed computing.

The fourth element in our vision focuses on the nature and time scale for human
creativity. Research on computing tools, algorithms and new models of physical sys-
tems pay significant dividends but such research takes time and commitment par-
ticularly when associated with changes in the underlying computing paradigm.

This fourth element as well as the final three are themes that are also dealt with
in the HPCC report.

Our recommendations follow directly from this vision.
Recommendations

• Continuing Need for the Centers Program

• Specific infrastructure Characteristics for Leading-Edge Sites

• Partnering for a More effective National Infrastructure

• Competition and Evaluation

• Support of Research at the Centers

• Allocation Process for Computer Service Units

• NSF Leadership in Interagency Planning

The first recommendation is that NSF should continue to provide access to the
high-end. This infrastructure is central to maintaining U.S. researchers among the

leaders in long term fundamental research. The Centers Program is needed in our
view to do this effectively. As new advanced computer hardware becomes commer-
cially available, scalability will continue to be a challenge for the foreseeable future.

Each new generation of computing power brings with it vast new opportunities
for advances in fundamental understanding and synthesis of complex — often inter-
disciplinary — research aieas. The take-home message from our report is that lead-
ing edge computing has and will continue to make significant contributions to the
advance of science and engineering.

Our second recommendation is that NSF needs to support a few "Leading Edge"
sites. Sites that have a balanced set of high-end hardware capabilities coupled with
appropriate staff and software. Raw processor speed or compute power is not suffi-
cient to achieve the balanced high-performance systems needed for paradigm shift-
ing research.

In our third recommendation we focus on partnerships. Partnerships will be im-
portant for the future of the centers program. The strategic coupling between lead-
ing-edge sites and research centers and university laboratories is a significant new
change that we are recommending. These partnership sites will provide better cou-
pling to computer scientists developing new tools and software — this coupling will
be particularly important when their research on mid-range systems reaches the
point when it needs to examine scalability issues that become manifest only on the
leading-edge systems. But the impact of this research will extend beyond computer
science. The partnership sites will also provide more cost effective and diverse plat-
forms for early design and testing of applications software prior to the stage where
access to a leading-edge system is required.

It has long been recognized that some portion of the resources at the NSF Centers
was being used for jobs that did not require access to leading-edge systems. This
is particularly true during the startup, design and test stages of a project — even for
projects with significant overall computational requirements. With the development
of mid-range systems that are upwardly compatible with the leading-edge systems,
it should now be possible to make more effective use of the leading-edge systems
by reserving them, as much as possible, for computations that require that unique
level of capability. The resources on these mid-range systems should also be used
to obtain preliminary data on software performance that would guide the allocations
committees in their review of requests for access to leading-edge systems.

Recommendation four deals with the need for a new competition and for evalua-
tion of the overall program on a regular basis. We believe that a new competition
is the best way to proceed. A new competition can be fair to all and stimulate the
greatest levels of creativity in proposal preparation.

Given the rapid advance of computing hardware and communications technology,
there will be a continuing need to review the centers program. NSF will need to
carry out regular annual reviews, as in the past, and probably a full program review
in about five years.

Recommendation five deals with the support of research at the centers. Centers
need to have high levels of expertise. This requires significant staff involvement
with research, but not direct funding of independent research by the Centers Pro-
gram.

Recommendations six and seven are suggestions to NSF management on leader-
ship issues that the Task Force believes that management needs to focus on, but
I do not believe that they are central to these hearings.

Although we began with the work of earlier reports, we did not accept their con-
clusions or recommendations uncritically. We carried out our own analysis, particu-
larly of the merit review process for allocating resources at the current centers, an
analysis of the educational benefits of the program, and we also sought significant
input from the community, including several key NSF Advisoryy Committees, on the
benefits of the program.

In preparing for is hearing I have been able to talk with most of the members
of the Task Force to confirm their enthusiasm for the manner in which NSF man-
agement is moving forward with the program announcement and the receipt of pro-
posals. All indicated that they were pleased with the progress that has been made
up to this point.

Thank you for your attention. I will be happy to answer questions.

ATTACHMENT 1

bX)

CIh

Cd C^ c/2

.Jri 'U ^

Cd (D (D

:^ ^ u

C/3

;-i

<1>

-4— >

^— »

4->

OX) cf^

PARTNERSHIPS FOR
ADVANCED
COMPUTATIONAL
INFRASTRUCTURE

Program Solicitation

DIRECTORATE FOR COMPUTER AND INFORMATION SCIENCE
AND ENGINEERING

PREPROPOSAL DEADLINE: April 15, 1996
PROPOSAL DEADLINE: September 1, 1996

S?^ NATIONAL SCIENCE FOUNDATION

Program Solicitation
Partnerships for Advanced Computational Infrastructure

Overview

NSF support of high performance computing has played a
major role m advancing science and engineering research and in
enabhng U. S. world leadership in computational science and
engineering. The NSF Supercomputer Centers program has
served as a cornerstone of these advances by expanding the use
of high-end computing in new disciplines, facilitating the accep-
tance of computation as a full partner in scientific research, and
facilitating the education of a new generation of computational
scientists and engineers in support of that shift.

This program solicitation for the Partnerships for Advanced
Computational Infrastructure program, builds on and replaces
the current NSF Supercomputer Centers program established in
1985, and focuses on taking advantage of newly emerging op-
portunities in high performance computing and communica-
tions. This new program will provide flexibility, both to adapt to
rapidly evolving circumstances and to meet the need for high-end
compulation, in order to enable continued world leadership in
computational science and engineering. The program will pro-
vide access for researchers to high performance computing sys-
tems as its core, with associated highly trained staff and
researchers necessary to develop and optimize their use The
emergence of scalable parallel systems, high performance net-
working and high bandwidth, large capacity mass storage sys-
tems creates the opportunity for a national infrastructure
consisting of a number of geographically distributed sues
strongly coupled to high-end computational resources and to
each other via high-speed communication networks.

NSF envisions an Advanced Computational Infrastructure
consisting of one or more leading-edge sites together with coop-
erating partners. Leading-edge sites are expected to maintain
high-end hardware systems that are one to two orders of magni-
tude more capable than those typically available at a major
research university. These systems should be balanced in terms
of processor speed, memory, and storage systems, and should be
accompanied by appropriate staff, software and high speed com-
munications capability. The partners will, in the aggregate,
complete the overall infrastructure by, among other things,
(a) facilitating research and experimentation with new hardware
and software, including appropriate support technologies such as
visualization and mass storage, (b) providing scalable resources
for applications and applications development that can be best
done on mid-level systems, (c) providing access to unique ex-
perimental systems and facilities, and (d) promoting education
and training.

Background

Several reports analyzing the current Supercomputer Centers
Program and making recommendations for future computational
infrastructure have appeared recently. Most directly relevant to
this solicitation is the Report of the Task Force on the Future of

the NSF Supercomputer Centers Program, chaired by Dr Ed-
ward Hayes, September, 1995, which discusses the history, cur-
rent context, and rationale for NSF support of high performance
computational infrastructure for the science and engineering
research communities.

Two other reports. From Desktop to Teraflop: Exploiting the
U.S. Lead in High Performance Computing, chaired by Dr.
Lewis Branscomb, October, 1993, and Evolving the High Per-
formance Computing and Communications Initiative to Support
the Nation's Information Infrastructure, chaired by Drs.
Frederick Brooks and Ivan Sutherland, National Research Coun-
cil, February, 1995 provide additional background and discuss
the overall impact of these programs in the context of the larger,
federally supported High Performance Computing and Commu-
nications program.

These reports are available on the World Wide Web via URL
http://www.cise.nsf.gov/.

Hard copies of the first two reports may be obtained by
sending e-mail to bmelvin@nsf gov. The NRC report is available
from the National Research Council, 2101 Constitution Ave.
N.W., Washington. DC 20418.

Mission

As part of its strategic plan. NSF in a Changing World, a key
NSF goal IS to enable the United States to uphold a position of
world leadership in science and engineering research and educa-
tion. In order to maintain world leadership in computational
science and engineering, NSF intends to create an advanced
national computational infrastructure whose overall mission is
to:

• provide, facilitate, and enhance access to high performance
computational infrastructure for the U. S. academic, scien-
tific, and engineenng communities by partnering with
universities, states, and the private sector;

• promote vigorous early use of experimental and emerging
high performance computational and associated commu-
nications technologies that offer high potential for advanc-
ing science and engineering;

• enable the effective use of such infrastructure and tech-
nologies through education, training, consulting, and re-
lated support services, including appropriate software
development, experimentation, and support;

• foster interdisciplinary research in science and engineer-
mg;

• facilitate the development of the intellectual capital re-
quired to maintain world leadership in computational sci-
ence and engineering; and

• broaden the base for the nation's advanced computational
and communications mfraslructure.

Purpose of this Solicitation

This solicitation calls for Innovative proposals to provide a
national computational infrastructure that will address the mis-
sion stated above. Guidance for participating in this solicitation
IS provided below.

Since the Partnerships for Advanced Computational Infra-
structure Program will primarily support academic research, it is
expected that proposals will come from, and the partnership's
cooperative agreement will be with, a U. S. academic institution.
However, a successful proposal must involve multiple partners
who can be. but are not limited to:

• universities, including research groups within universities;

• NSF-funded Centers and facilities, such as. Science and
Technology Centers. Supercomputer Centers, Engineering
Research Centers, and Industry-University Cooperative
Research Centers;

• research and educational consortia, organizations, and
groups;

• regional and stale-supported high-performance computing
centers;

• private sector organizations; and

• national laboratories.

A full description of organizations and individuals who are
eligible to submit proposals to NSF and the conditions under
which they may compete, are given in the NSF Grant Proposal
Guide (GPG) (NSF 95-27). Single copies of this brochure are
available at no cost from the NSF Forms and Publications Unit,
(703) 306- 1 1 30, or via e-mail: pubs(a'nsf gov. Brochures are also
available electronically through NSF's Science and Technology
Information System (STIS) and on the World Wide Web at URL
htlp://www.nsf.gov/.

Review Process and Criteria

The review process will involve preproposals and full propos-
als, both of which will be reviewed by external panels using
criteria discussed below. Panelists will be chosen from a variety
of sectors including academia, the private sector, national labo-
ratories, and other government agencies. Site visits are antici-
pated for finalists in the competition in order to clarify issues
raised during the reviews, and to explore additional matters as
needed. A summary panel, consisting of the chairs of the site visit
teams and the chair of the final proposal review panel, will meet
after the site visits to formulate final recommendations for a
balanced program.

As with all proposals to NSF, these proposals will be evaluated
using the four standard review criteria: (I) the quality of the
proposed scientific effort. (2) competence of the investigators.
(3) relevance of the research, and (4) impact on the infrastructure
of science and engineering. However, for this particular solicita-

tion, impact on infrastructure and competence of investigators
will be especially important. Therefore preproposals and final
proposals for this program will be subject to four, more specific,
criteria dealing with the effectiveness of the overall partnership
in addressing the program's mission, the quality of the individual
partners, management, and financial leverage. While these more
specific criteria are listed in priority order (with the first being
the most important), all will be considered by the panels in
arriving at recommendations. Finally, the summary panel will be
asked to advise NSF on what each proposed partnership can
contribute to the technical diversity and balance of the program
as a whole. In assessing the degree to which each preproposal
and proposal satisfies a particular criterion, the reviewers will be
asked to consider each of the following points.

1 ) The degree to which the proposed partnership addresses the
overall mission statement and demonstrates that it will be able
to:

• provide the physical and human infrastructure needed to
fulfill a national leadership role with an intellectual envi-
ronment that can foster world leadership in computational
science and engineering research across all NSF disci-
plines;

• maintain balanced computational resources, including at
least one site with capabilities one to two orders of magni-
tude greater than that typically available at a major re-
search university;

• enable interdisciplinary partnerships among the academic
computer science, mathematics, and computational sci-
ence research communities;

• support and develop software enabling computational sci-
ence and engineering;

• effect outreach and technology transfer to a heterogeneous
user base;

• support cooperative relationships with hardware and soft-
ware vendors;

• foster the integration of research, education, and training
in computational science and engineering;

• promote the advancement of kindergarten through under-
graduate science and engineering education generally; and

• provide world-class leadership through a diverse and syn-
ergistic set of partners.

2) The degree to which the proposed individual partners demon-
strate expertise in at least one of the following:

• providing access to. and support for. high performance
computing, including appropriate supporting technolo-
gies;

• conducting world-class research in computational science
and engineering, computer science and engineering, or an
appropriate related field;

• promoting vigorous early use of experimental and emerg-
ing scalable computing, communication, or mass storage
technologies;

• providing training, education, and outreach to the research
and educational communities and to the private sector: and

• supporting cooperative efforts across multiple intellectual
and/or institutional sectors.

3) The quality of the proposed management of the partnership
including:

• clear plans for the close coordination and management of
the partnership, including the computational, communica-
tions, and intellectual resources of the partners; and

• plans for measuring the partnership's success in meeting
its goals.

4) The degree of financial leverage in the proposed program,
including cost sharing. This can include personnel provided
by institutions and vendors, state and local support, facilities,
vendor discounts beyond normal educational discounts, soft-
ware, etc.

In addition, the final selection by NSF will consider what each
partnership can contribute to the technical diversity and balance
of the program as a whole.

Funding Levels and Duration of Awards

NSF funding for the current NSF Supercomputer Centers
Program totals about $65 million per year. Funding levels for the
new program will depend on expected overall funding levels for
NSF at the time of the award(s). NSF funding will enable active
participation in the partnership and create a broad base support-
ing the national computational infrastructure for research in
science and engineering. The number and size of awards will be
based on the quality and potential impact of the proposed part-
nerships, and on the availability of funds. Each partnership will
be funded through a single cooperative agreement specifying the
level of support for each partner, as proposed by the partnership,
and negotiated with NSF before the award. The cooperative
agreement for the partnerships is expected to cover a five-year
period beginning in fiscal year 1998.

Phase in of a new partnership is expected to occur over a one
year period. Partnership plans and proposed budgets must be
prepared with enough flexibility to allow the phasing in of
partnership resources during the first year, as negotiated with
NSF. To provide flexibility in meeting special needs, funding can
be provided for partnerships having a membership that may
change over the duration of the cooperative agreement; thus,
proposals involving partners for a shorter period are encouraged,
where appropriate. Funding will be provided in yearly incre-
ments subject to successful annual review of program plans,
performance, and the availability of funds.

To provide a partnership with the flexibility to respond to an
unusual opportunity, completion of a partner's task, changes or
relocation of a critical person, etc., the partnership may propose
(at any time, subject to review), a change in the number or

composition of its partner sites. Normally such proposals will be
pan of the annual program plan review process and would be
considered along with other substantive issues.

The partnerships will be reviewed annually by an external
partnership review panel, with site visits in the second year. In
the fourth year of the program, NSF will conduct a full inde-
pendent review of the overall program. Subject to the needs of
the scientific and engineering community as determined by this
review; the acceptable annual progress and performance reviews
of the partnerships; and the availability of funds, NSF may invite
renewal proposals, with the intent of extending successful part-
nerships for an additional five-year period.

At the end of ten years, this program will be "sunset" . During
the eighth year there will be a full and independent review to
determine the anticipated future needs of the academic science
and engineering community

Program IVIanagement

Each partnership will offer a variety of resources. In addition
to computational resources, there might be visualization re-
sources, such as access to a virtual reality environment, mass
storage resources located at a single site, discipline-specific
knowledge repositones, software development resources, code
porting and optimization resources, discipline-specific subrou-
tine and program libraries, performance instrumentation tools
and libraries, generic subroutine libraries such as linear algebra
codes, and educational and training resources.

The individual partnerships are being asked to propose man-
agement plans, allocation mechanisms, and means for obtaining
external advice. However, NSF will require that the partnerships
develop a single mechanism for allocation of the use of the bulk
of the program's resources, independent of where they are lo-
cated

Budget

Funds may be requested to support scientific and technical
staff essential for systems programming, software development,
research and user services, coordination and conduct of educa-
tion, training and outreach activities, and participant support;
training materials and software; networking equipment and op-
erating costs; and computer systems, indirect costs and other
appropriate costs associated with the overall project. Proposers
should Itemize costs on the NSF Budget Form 1030 and include
additional supporting documentation Proposed cost sharing
should be recorded on Line M of the Budget Form 1030.

Cost Sharing

Cost sharing may be in cash or in-kind from any private or
non-federal source. The estimated value of any in-kind contribu-
tions should be included and an explanation of the source, nature,
amount, and availability of any proposed cost sharing should also
be provided. Cost sharing must occur during the award period
and must be consistent with OMB Circular A- 1 10, Section .23.
Cost sharing specified in the proposal will be referenced and
included as a condition of the award.

Preproposal Guidelines

In order to be eligible to submit a formal proposal, the institu-
tion must have submitted, by April 15, 1996, a preproposal that
was reviewed, or been named as a partner on a preproposal that
was reviewed. Preproposals will be reviewed by external panels
using the criteria listed above. The intent of the preproposal
round is to give the community an opportunity to put forth
diverse ideas, provide information that will assist in developing
a stronger proposal, and point out preproposals that are not
expected to be competitive. Review comments from the prepro-
posal round will be supplied about the middle of May 1996 to
the proposers in order to provide feedback for the final proposal
preparation. Multiple preproposals from the same proposing
organization involving different partners, funding levels, and
objectives are allowed Neither institutional approvals nor signa-
tures are required at this stage; however, proposals should follow
any applicable institutional guidelines for preproposals.

Preproposals should be mailed to;

Partnerships in Advanced Computational Infrastructure
NSF, Room 1 1 22
4201 Wilson Boulevard
Arlington, VA 22230

Preproposals must be received by NSF no later than April 15,
1996; or postmarked no later than five (5) days prior to the
deadline date; or sent via commercial overnight mail no later than
two (2) days prior to the deadline date, to be considered for
review.

Preproposals submitted in response to this solicitation must be
prepared and submitted in accordance with the guidelines pro-
vided in the NSF Gram Proposal Guide. (GPG ) (NSF 95-27); in
particular, page formatting requirements given in Chapter 2,
Section C, will be strictly enforced and preproposals not com-
plying will be returned without review. Preproposals should
allow reviewers to address each of the evaluation criteria listed
above.

Preproposals (15 copies) must include:

• A cover page listing the Pi's e-mail address and FAX
number, and all participating sites and organizations;

• An executive summary of 2-3 pages;

• A main body of 10 single-spaced pages, to include:

- goals and objectives;

- a description of the proposed partnership activities;

- an overview of the partnership, its participants, and
plans for closely coordinating the participants;

- a staffing description and summary of the key person-
nel at each site;

- a list of the expected accomplishments of the partner-
ship over the five-year period of the award;

- a proposed management plan for the partnership, in-
cluding mechanisms for making policy and planning
decisions, and obtaining advice from the community;

- a proposed method for allocating and closely coordi-
nating the computational and intellectual resources of
the partnership; and

- a plan for evaluating the success of the partnership.

• Adescription of proposed hardware, networking, and soft-
ware, including planned upgrades;

• Key planned software and algorithm developments, if
applicable;

• Cost sharing projections, both cash and in kind;

• A non-binding draft budget for each year (the proposed
funding level for each partner should be specified);

• An appendix containing one to two page vita including the
most relevant publications for the key personnel; and

• No additional appendices are allowed.

The preproposals will be reviewed by external panels. It is
anticipated that preproposal review will be completed and feed-
back provided to the proposers by mid-May 1996.

Proposal Guidelines

To be eligible to submit a formal proposal, the institution must
have submitted a preproposal that was reviewed, or have been a
partner on a preproposal that was reviewed.

Proposals should be mailed to:

Partnerships in Advanced Computational Infrastructure
NSF Room 1122
4201 Wilson Boulevard
Arlington, VA 22230

Proposals must be; received by NSF no later than September I ,
1996; or postmarked no later than five (5) days prior to the
deadline date; or sent via commercial overnight mail no later than
two (2) days prior to the deadline date, to be considered for
review.

Proposals submitted in response to this solicitation must be
prepared and submitted in accordance with the guidelines pro-
vided in the NSF Grant Proposal Guide (GPG) (NSF 95-27); in
particular, page formatting requirements given on page 3, Sec-
tion C, will be strictly enforced and proposals not complying will
be returned without review. Proposals should allow reviewers to
address each evaluation criterion listed above.

Proposals (15 copies) should include the following:

• A cover page listing the Pi's e-mail address and FAX
number, and all participating sites and organizations;

• An executive summary of 2-3 pages;

• A main body of 40 single-spaced pages, to include:

- goals and objectives;

- a description of the proposed partnership activities;

- an overview of the partnership, its participants, and
plans for closely coordinating the participants;

- a staffing description and summary of the key person-
nel at each site;

■ a hst of the expected accomplishments of the partner-
ship over the five-year period of the award;

- a proposed management plan for the partnership in-
cluding mechanisms for making policy and planning
decisions, and obtaining advice from the community;
a proposed method for allocating and closely coordi-
nating the computational, and intellectual resources of
the partnership; and

- a plan for evaluating the success of the partnership.

• A description of proposed hardware, networking, and soft-
ware, including planned upgrades;

• Key planned software and algorithm developments, if
applicable;

• Cost sharing proposed, both cash and in kind;

• Abudget for each year(the proposed funding level for each
partner should be specified);

• An appendix containing one to two page vita including the
most relevant publications for the key personnel;

• An appendix containing letters that commit actual re-
sources such as funds, hardware, people, space, etc. from
the participating institutions, vendors, states, etc. and

• No additional appendices are allowed.

Proposals will be reviewed by external panels. While there will
be some overlap between the panels reviewing the preproposals
and final proposals, the panelists will be instructed that any
recommendations based on the preproposal review were not
intended to be binding on the proposer Recommendations and
reviews from the preproposal round will not be available to the
final proposal panel, and should not be considered in reviewing
the final proposals. Site visits are anticipated for finalists in the
competition in order to clarify issues raised during the reviews,
and to explore additional matters as needed.

Panel reviews are expected to be completed by October, 1996,
and site visits will occur before the end of the 1 996 calendar year.
Awards should be announced in the Spring of 1997.

Inquiries related to the program should be sent via e-mail to
PACI(s)nsfgov

or they may be sent by US mail to:

Partnerships in Advanced Computational Infrastructure
NSF, Room 1122
4201 Wilson Boulevard
Arlington, VA 22230

GENERAL INFORMATION

The Foundation provides awards for research in the sciences
and engineering. The awardee is wholly responsible for the
conduct of such research and preparation of the results for
publication. The Foundation, therefore, does not assume respon-
sibility for the research findings or their interpretation.

The Foundation welcomes proposals from all qualified scien-
tists and engineers and strongly encourages women, minorities,
and persons with disabilities to compete fully in any of the
research related programs described here.

In accordance with federal statues, regulations, and NSF poli-
cies, no person on grounds of race, color, age, sex, national
origin, or disability shall be excluded from participation in. be
denied the benefits of, or be subject to discrimination under any
program or activity receiving financial assistance from the Na-
tional Science Foundation.

Facilitation Awards for Scientists and Engineers with Dis-
abilities (FASEDl provide funding for special assistance or
equipment to enable persons with disabilities (investigators and
other staff, including student research assistants) to work on NSF
projects. See the program announcement ( NSF 9 1 -54 ) or contact
the program coordinator at 306-1636.

Privacy Act and Public Burden. The information requested
on proposal forms is solicited under the authority of the National
Science Foundation Act of 1950, as amended. It will be used in
connection with the selection of qualified proposals and may be
disclosed to qualified reviewers and staff assistants as part of the
review process; to applicant institutions/grantees to provide or
obtain data regarding the application review process, award
decisions, or the administration of awards; to government con-
tractors experts, volunteers, and researchers as necessary to
complete assigned work; and to other government agencies in
order to coordinate programs. See Systems of Records. NSF 50,
Principal Investigators/Proposal File and Associated Re-
cords, and NSF-51. 60 Federal Regislei 4449 (January 23,
1995) Reviewer/Proposal File and Associated Records, 59
Federal Register 8031 (February 17, 1994). Submission of the
information is voluntary. Failure to provide full and complete
information, however, may reduce the possibility of your receiv-
ing an award.

Public reporting burden for this collection of information is
estimated to average 120 hours per response, including the time
for reviewing instructions. Send comments regarding this burden
estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Herman G.
Fleming. Reports Clearance Officer. Contracts, Policy, and
Oversight, National Science Foundation, 4201 Wilson Boule-
vard, Arlington, VA 22230.

The National Science Foundation has TDD and FIRS capabil-
ity, which enables individuals with hearing impairment to com-
municate with the Foundation about NSF programs,
employment, or general information. To access TDD dial (703)
306-0090; for FIRS, 1-800-877-8339.

6ETTIN6 NSF INFORMATION AND PUBLICATIONS

The National Science Foundation (NSF) has several ways for the public to receive information
and publications. Electronic or printed copies of the NSF telephone directory, abstracts of
awards made since 1989, and many NSF publications are available as described below. To
access information electronically, there is no cost to you except for possible phone and Internet
access charges. Choose the method of access that matches your computer and network tools. For
general information about Internet access and Internet tools, please contact your local computer
support organization.

WORLD WIDE web:
NSF HOME PACE

The World Wide Web (WWW) syslem
makes it possible to view text material
as well as graphics, video, and sound.
You will need special software (a "web
browser") to access the NSF Home
Page. The URL (Uniform Resource
Locator) is http://www.n5f.g0v/.

INTERNET «OPHER

The Internet Gopher provides access to
information on NSF's Science and
Technology Information System
(STIS) through a series of menus. To
access the Gopher, you need Gopher
client software; the NSF Gopher server
Is on port 70 ofstis.nsf.gov.

ANONYMOUS FTP (FILE
TRANSFER PROCRAM)

Internet users who are familiar with

FTP can easily transfer NSF

documents to their local system for

browsing and printing. The best way

to access NSF information is to first

look at the index (file name:

index.txt). From the index, you can

select the files you need. FTP

instructions are:

■ FTPtostis.nsf.gov.

m Enter anonymous for the user name.

and your e-mail address for the

password.
• Retrieve the appropriate file (i.e.,

filename.ext)

E-MAIL (ELECTRONIC-MAIL)

To get documents via e-mail, send your
request to the Internet address
slisserve@nsf.gov The best way to
find NSF information is to request the
index. Your e-mail message should
read: get index.txt. An index with file
names will be sent to you. However if
you know the file name of the
document you want, your e-mail
message should read:
get <filenanie.ext>

E-MAIL MAILINC LISTS

NSF maintains several mailing lists to
keep you automatically informed of
new electronic publications. To get
descriptions of the mail lists and
instructions for subscribing, send your
request to: slisserve@nsf.gov. Your
message should read: get stisdirm.txt.

ON-LINE STIS

NSF's Science and Technology
Information System (STIS) is an
electronic publications dissemination
system available via the Internet (telnet
to slis.nsf.gov), you will need a VTIOO
emulator. The system features a full-
text search and retrieval software
(TOPIC) to help you locate the
documents. Login as public and follow
the instructions on the screen.

To get an electronic copy of the "STIS
USERS GUIDE." NSF 94-10. send an
e-mail request to: stisserve@nsf.gov.
Your message should read:
get NSF9410.txt. For a printed copy of
the "STIS USERS GUIDE," see
instructions "How To Request Printed
NSF Publications."

NON-INTERNET ACCESS
VIA MODEM

If you do not have an Internet
connection, you can use remote login
to access NSF publications on NSF's
on-line system, STIS You need a
VTIOO terminal emulator on your
computer and a modem.

• Dial 703-306-0212,

• choose 1 200, 2400, or 9600 baud,

• use settings 7-E- 1 , and

• login as public and follow the on-
screen instructions.

NSF 95-64 (Replaces NSF 94-4)

HOW TO REQUEST PRINTED
NSF PUBLICATIONS

You may request printed publications
in the following ways:

• send e-mail request to:
pubs@nsf.gov

m fax request to: 703-644-4278

■ for phone request, call: 703-306-
1130 or Telephonic Device for the
Deaf (TDD 703-306-0090)

■ send written request to:

NSF Forms and Publications Unit
4201 Wilson Boulevard
Room P- 15
Arlington, VA 22230

When making a request, please include
the following information:

• NSF publication number;

• number of copies; and

• your complete mailing address.

QUESTIONS ABOUT NSF
PUBLICATIONS. PROCRAMS/
ETC

Contact the NSF Information Center if
you have questions about publications,
including publication availability,
titles, and numbers. The NSF
Information Center maintains a supply
of many NSF publications for public
use. You may:

• visit the NSF Information Center,
located on the second fioor at 4201
Wilson Blvd., Arlington, Virginia.;
or

• call the NSF Information Center at
703-306-1234; or 703-306-0090 for
TDD; or

• send e-mail message to
info@nsf.gov

QUESTIONS ABOUT THE
ELECTRONIC SYSTEM

Send specific, system-related questions
about NSF electronic publication
services that are not answered in this
fiyer, to webmaster@nsf.gov or call
703-306-0214 (voice mail).

Report of the Task Force on the Future of the NSF Supercomputer Centers Program

Executive Summarv

The NSF supported Supercomputer Cen-
ters have played a major role in advanc-
ing science and engineering research.
They have enabled collaboration among
academic, industrial and government re-
searchers on the solution of problems re-
quiring demanding computational and
visualization tools. In the 10 years of
their existence, the Centers have fostered
fundamental advances in our under-
standing of science and engineering, ex-
panded the use of high-end computing in
new disciplines, facilitated the major
paradigm shift of the acceptance of com-
putational science as a full partner in the
.scientific method, and facilitated the edu-
cation of a new generation of computa-
tional scientists and engineers in support
of that shift. These statements are docu-
mented in the body of the report as well
as in its appendices.

Having accomplished so much in the la.st
decade, it is natural to ask what the future
role of the NSF should be in high-end
computing. In October 1994. the Na-
tional Science Board approved two-year
contmuation funding for the Supercom-
puter Centers. This provided time for the
director of NSF to appomt this Task
Force to analyze the alternatives. The
possibilities considered include continua-
tion, restructuring, or phasing-out of the
current program, as well as creation of
alternative models.

The Task Force believes that the future
for computational science and engineer-
ing can be as bright or even brighter than
in the past decade. If we seize the op-
portunity, over the next decade we can
make major progress on multiple fronts.

There will be

• opportunities for exciting applica-
tions of our nation's exponentially
increasing computational capacity,
for example:

- more complete models, and hence
deeper understanding of physical
systems by moving to three and
higher dimensions;

- progress in computational tools to
aid drug and protein design;

- computational predictions of sci-
entifically and commercially sig-
nificant materials;

- multidiscipiinary models of
physical systems (e.g., combining
fluid dynamics and electromag-
netic models of the heart);

- increased interconnectivity of su-
percomputers and high impact in-
.strumentation; and

- models of anatomical and
physiological proces.ses leading to
new insights of benefit to human
health.

• more quantitative computational
results in unanticipated areas.

• more explosive growth of com-
munications as a component of the
computational .science and engi-
neering paradigm; and. impor-
tantly.

• continued progress in the tools and
methods for developing code that
is both portable and yet takes ad-
vantage of unique parallel archi-
tectures.

These advances will not automatically be-
come available to American researchers.
To position the U.S. academic commu-
nity to participate in the exciting research
possibilities enabled by these develop-
ments, the Task Force has the following
recommendations leading to a restruc-
tured Centers program.

September 15, 1995

Report of the Task Force on the Future of the NSF Supercomputer Centers Program

Executive Summary

In order to maintain world leadership
in computational science and engi-
neering, NSF should continue to
maintain a strong, viable Advanced Sci-
entific Computing Centers program,
whose mission is:

• providing access to high-end
computing infrastructure for the
academic scientific and engi-
neering community;

• partnering with universities,
states, and industry to facilitate
and enhance that access:

• supporting the effective use of
such infrastructure through
training, consulting, and related
support services:

• being a vigorous early user of
experimental and emerging high
performance technologies that
offer high potential for advanc-
ing computational science and
engineering;

• facilitating the development of the
intellectual capital required to
maintain world leadership.

NSF should assure that the Centers
program provides national "Leading-
edge sites" that have a balanced set of
high-end hardware capabilities, coupled
with appropriate staff and software,
needed for continued rapid advance-
ment in computational science and en-
gineering.

NSF, through its Centers program,
should assure that each leading-edge
site is partnered with experimental fa-
cilities at universities, NSF research
centers, and/or national and regional
high performance computing centers.
Appropriate funding should be provided
for the partnership sites.

NSF should announce a new com-
petition of the High Performance Com-

puting Centers program that would
permit funding of selected sites for a
period of five years. If regular reviews
of the Program and the selected sites
are favorable, it should be possible to
extend initial awards for an additional
five years without a full competition.

The Centers program should continue
to support need-based research in sup-
port of the program's mission, but
should not provide direct support for
independent research.

NSF should increase the involvement of
NSF's directorates in the process of al-
locating service units at the Centers.

NSF should provide leadership in
working toward the development of in-
teragency plans for deploying balanced
systems at the apex of the computa-
tional pyramid and ensuring access to
these systems for academic researchers.

These recommendations are designed to
set the Centers program on a new course
that builds on its past successes, yet shifts
the focus to the present realities of high-
performance computing and communica-
tions, and provides flexibility to adapt to
changmg circumstances. It is our expec-
tation, that at current NSF budget levels
and absent new outside resources, there
will be a reduction in the number of lead-
ing-edge sites to effect the benefits of the
Task Force recommendations.

In developing these recommendations,
the Task Force obtained extensive input
from academic, government, and indus-
trial leaders: visited Centers and sought
written input from the community.
Some of this input is included as appen-
dices, and the complete input is available
on the Internet. The issues are complex
and there are many strongly held opin-
ions on the purpose, execution, and value
of the program. The Task Force has tried
to hear and understand all of the input.

September 15, 1995

Report of the Task Force on the Future of the NSF Supercomputer Centers Program

Executive Summarv

but in the end has, of necessity, formed
its own judgment of what is best for the
country. This report attempts to explain
that judgment.

The report begins with a history of the
Centers and how they fit into the nation's
high performance computing infrastruc-
ture.

The .second section attempts to identify
factors the Task Force thinks are impor-
tant in the evaluation process, including
staff involvement in research, size of the
u.ser ba.se, scientific discipline of the us-
ers, funding leverage, industrial partner-
ships, multidi.sciplinary activities, re-
source availability, and education.

The "hard issues" surrounding the Cen-
ters, particularly those not adequately dis-
cussed in previous reports, are discussed
in the third section. This .section exam-
ines such issues as: sunsetting the Cen-
ters: industrial u.se; effect of "free" com-

puter cycles on the market: the total need
for high-end computing; quality of the
science; role of other centers; technology
and computer industry trends; and the
role of the Centers in the larger federal
and international context.

Section four examines five options for a
Centers program, ranging from the cur-
rent system to termination of the pro-
gram. Other options include partnership
centers with stronger links between lead-
ership centers and university or state fa-
cilities; a single partnership center; and
disciplinary centers along the lines of the
National Center for Atmospheric Re-
.search. The pros and cons of each option
are discussed.

The fifth section of the report discusses
future directions and priorities for the
Centers program. The final section re-
states and explains each of the seven spe-
cific recommendations designed to sup-
port the Task Force vision for the future.

September 15, 1995

High Performance Computing Infrastructure and Accomplishments

October 1994
Table of Contents

INTRODUCTION 1

IMPORTANT TECHNOLOGY ACCOMPLISHMENTS 1

hi(;h performance computing i

Supcrcunipuler Usage at NSF Centers 2

Aicliuectuies and Vendors 2

Center Program Chronology 2

National Access to Vector Multiprocessors 4

Aclueving Production Parallelism 4

Early Migration to the UNIX Operating System 4

Early Access to Massively Parallel Computers 4

Superlincar speedup on heterogeneous processors 5

Workstation Clusters 5

PORTABLE PARALLEL PROGRAMMING TOOLS 6

Prototype Parallel Programming Environments 6

Extensions of PVM 6

Scalahle Libraries <i

STORAGE TECHNOLOGIES 6

AFS-Establishing a National File System.. 7

HDF-Creating a Standard File Formal 7

Migrating to a Standard Archiving Software 7

Development ol high-densily magnetic media 8

NET\VORKIN(; 8

Evolution ol NSFNET 9

High Performance LANs 4

Gigabit Testheds 10

Secure Networks H'

New Science Enabled by Networks— Teleniicroscopy 10

Nil Testbeds II

VISUALIZATION AND VIRTUAL REALITY 12

DcNclopincnt of Scientitic Visualization 12

Virtual Realit) Impacts Industrial Design 1 3

DcvelopnieiU of Immersive Science Projects 14

Virtual Reality over ATM networks 14

Alpha Shapes. Biomolecules. and Cosmology 14

DKilTAL LIBRARIES AND INFOSERVERS 14

Digital Libraries 15

Scalable Information Servers 15

The Rise ol the Mosaic/WWW Information Infrastructure 15

DESKTOP SOFTWARE 15

Cunneclivitv Tools 16

High Performance Computing Infrastructure and Accomplishments

Collaboration Tools 16

Graphics Tools 16

Scientist' s Workbench 17

ACCOMPLISHMENTS IN EDUCATION AND OUTREACH 17

EDUCATION 17

Researchers and Students 17

Supercomputer Centers Educational Activity Support Summary 17

Outreach to Educators 18

OUTREACH 19

Application ot Scientific Compulation and Visualization to Industrial Production 20

Impact on Vendors of High Performance Computini! Equipment 20

Stimulation of New, Computationally Dependent Ventures 21

Development of Nationally Valuable Reservoirs of Skill 22

Community Service 22

IMPORTANT SCIENCE AND ENGINEERING ACCOMPLISHMENTS 22

SUMMARIES OF COMPUTATIONALLY INTERESTING PROBLEMS IN THE NSF
CENTERS PROGRAM BY THE NATIONAL SCIENCE AND ENGINEERING COM-
MUNITIES: 22

QUANTUM PHYSICS AND MATERIALS 23

Phase Transition in QCD 23

Phase Transitions of Solid Hydrogen 23

Prediction of new Nanomalerials 23

Theory of High Temperature Superconductors 24

Magnetic Materials 24

Understanding Glass 24

BIOLOCY AND MEDICINE 25

Crystallography 25

Folding Proteins using Artificial InlelligcnLe 25

Protein Kinase solution 26

Molecular Neuroscience-Serotonin 26

Molecular Neuroscience-Aceiylcholinesierase 26

Kinking DNA ' 27

Antibody-Antigen Docking 27

Tuning Biomolecules to Fight Asthma 27

Virtual Spider and Artificial Silk 28

Heart Modeling 28

ENGINEERING 28

Ullra-high-strength Steels.. 28

Continuous Casting of Sleel 29

Beverage can design 29

Designing a Leakproof Diaper 29

Bone Transplant Bioengineering 29

Improving Performance with Riblels 29

Designing Belter Aircraft.. 30

Crash Testing Street Signs 30

High Performance Computing Infrastructure and Accomplishments

EARTH SCIENCES AND THE ENVIRONMENT 30

Deioxiticalion of Ground Water 30

Sage Grouse-Endangered Species and the US Army 31

Slorm modeling/forecasting 31

Los Angeles Smog 32

Upper Ocean Mixmg 32

Simulating Climate using Distributed Supercomputers 32

PLANETARY SCIENCES, ASTRONOMY, AND COSMOLOGY 32

Comet Collision with Jupiter 33

Discovery ol First Extrasolar System Planet 33

Building the BIMA Radio Telescope 33

Pulsar Searching and Discovery • 34

Accretion Disks Around Black Holes 34

Black Hole Collision Dynamics 34

Largest cosniological simulation 35

EVOLUTION OF THE METACENTER CONCEPT 35

RECOGNITION ACCORDED NSF SUPERCOMPUTER USERS AND PROJECTS 36
ACRONYMS 3 8

High Performance Computing Infrastructure and Accomplishments

Introduction

The NSF Centers Program was established
to provide access to high pert'ormance com-
puters (supercomputers) tor the broad Sci-
ence and Engmeermg Research Community.
The program has evolved from one com-
prising independent, competitive. ar>d similar
computer centers to one mcluding more co-
operative and diverse activities. Coordinating
the mission of tlie indi\iduai centers has in-
creased the diversity of computer architec-
tures available to the research community,
and has accelerated outreach to segments of
the community which had not before been
able to use the power of high performance
computers. At the same time, competition
between centers has been managed by NSF
and its advisory committees to the advantage
of the engineering and science communities
which the program was established to .serve.

Building on each center's tradition of pro-
viding a stable source of computer cycles for
a large community of .scientists and engi-
neers, the centers have evolved into a unique

resource to test which diverse computer ar-
chitectures best match the most demanding
problems posed by the community of uni-
versity researchers and to develop the neces-
sary supporting software and algorithms. For
example, this approach has enabled the cen-
ters to test which applications can be effi-
ciently served on newly developed systems
using clusters of the new generation of work-
stations that are now being introduced. Such
experiments are enabled b\ the open en\i-
ronment characteristic of the program.

During the first decade of the centers pro-
gram, major improvements in the delivery of
high performance computing have been de-
veloped, mainly by American computer re-
searchers and companies. But advances in
computing technology have been matched in
equal measure by improvements in computer
networking, and as a consequence the NSF
Centers have been a primary focus for accel-
erating the evolution of the Internet via
NSFnei. NREN. and the still evolving broad-
band width technology.

in this appendix, we provide detailed exam-
ples of the centers activities.

Important Technology Accomplishments

High Performance Computing

Originall> set up in 1985 to provide national
access to traditional supercomputers, the
NSF Centers have evolved to a much larger
mission. The Centers now offer a wide vari-
ety of high performance architectures from a
large array of American vendors. No longer
lUst adopting technology from the national
labs, the NSF Centers Program has become a
pioneering vanguard of technology — a
model for other agencies with a vested inter-

est in the high performance computing to
emulate. This today is dominated by research
efforts in software, with vital collaborations
with computer scientists, focusing on oper-
ating systems, compilers, network control,
mathematical libraries, and programming
languages and environments. The feedback to
the leading US vendors is increasing the use-
fulness of their product offerings to the sci-
entific and engineering communities, while
making them more competitive.

High Performance Computing Infrastructure and Accomplishments

Supercomputer Usage at NSF Centers

Fiscal Year

Active Users

Usage in CPU
Hours

1986

1,336

29,485

1987

3,299

95,751

1988

5,042

121,615

1989

5,967

165,960

1990

7,357

250,628

1991

7,723

361,073

1992

8,252

398,931

1993

7,735

910,088

1994

7,395

2,370,794

1995

6,601

4,590,606

(Usage IS 111 normalized CPU hours, based on comparative performance tests. The astoundmg leap m ca-
pacity in 1993 is mamly a result of the introduction of new computing architectures to solve the most
demanding of computational problems — the Gr2tnd Challenges. The slight decrease in the number of
users is the result of a concerted effort by the Centers to assist many of their users with small memory or
CPU-time requirements to meet their computational needs by the increasingly powerful workstations of
the mid-90's. The greatest benefactors of the increase m massively parallel cycles are the scientists and
engmeers addressmg the problems with the greatest computing demands)

Architectures and Vendors

The national community has been offered
access to a wide and frequently updated set of
high performance architectures since the begin-
ning of the NSF Supercomputer Centers Pro-
gram The current rate of change of the types of
architectures, and the number of vendors of-
i'enng them, is probably near an all time high.
We are in a period of ferment which the sci-
ence and engineermg communities sort out the
choices for finding an architecture that matches
theii various computational problems. A list of
architectures that the NSF Centers Program has
offered would include: single and clustered

high performance workstations or workstation
multiprocessors, minicomputers, graphics
supercomputers, mainframes with or without
attached processors or vector units, vector
supercomputers, and SIMD and MIMD mas-
sively parallel processors. Similarly, the list of
current vendors whose top machines have been
made available would include IBM, DEC,
Hewlett-Packard, Silicon Graphics, Sun Mi-
crosystems, Cray Research, Convex Com-
puter, Intel Supercomputer, Kendall Square.
Thinking Machines. nCUBE. Aliiant, Floating
Point Systems. ETA. Stellar, Ardent, and
Slardent.

High Performance Computing Infrastructure and Accomplishments

Center Program Chronology

Milestone or Event

Description

1986

? NSF Superaimputer Centers become
Dperational

Cornell Theory Center

National Center tor Supercomputing Applications

Pittsburgh Supercomputing Center

San Diego Superconiputer Center

lohn von Neumann Center

lyss-iywy

Renewal Re\ iew

1989

Renew 4 NSF Supercomputer Centers

CTC, NCSA, PSC, SDSC

1990-1492

ASL Ad\isor\- Committee report com-
pleted

strong recommendations tor adding parallel systems to
accompan\ the stable, production vector svstems

1991

,s \ ector Supercomputers operatmt;
3 Scalable parallel systems operatmg

IBM 3090 h processors (2)
Cray YMP 8 processors (2)
Alh'ant FX80 S processors
Cray 2S 4 processors
Cray YMP 4 processors
Con\'ex C240 4 processors
Intel iPSC/860 32 processors'''
TMC, CM2 32,000 processors (2|'^

1992

7 Vector Supercomputers operatmg
9 Scalable parallel systems operating

lomt Plannmg Initiated

Cray s\'stems remain

IBM ES9000/900 (Upgrade 3090)

Alhant 2800 (upgrade FX80)

Con\'e\ C3880 8 processors (Upgrade C240)
Intel iPSC/860 upgrade 64 processors)!
nCUBE2 128 processorst
TMC, CM5 512 processorst
KSRl 64 processorst
DEC Workstation Cluster (2)t
IBM Workstation Cluster
IBM PVS 32 processors

Initial meeting at SDSC Fall 92

1993

Joint Activities Began

hVector Supercomputers operating
13 Scalable parallel systems operating

Meeting at PSC
First |omt proposals to PPRP
First of Joint projects
Cray C90 16 processors
(other Crays, IBM, Convex stay)
Intel Paragon 400 processors upgrade t
KSRl upgrade 160 processorst
Hewlett Packard Cluster^
MasPar 2 16,000 processors^
IBM SPl 64 processorst
Cra\ T3D 5]2 processorst

1994

lomt Activities

4 Vector Supercomputers operating
14 Scalable parallel systems operating

.Meeting at CTC
Metacenter Regional Alliances - Mar 94
Expansion of |oint proiects
One YMP changed to C-90, others the same
IBM SP2 upgrade 512 processors upgrade SPlt
Cray T3D 512 processors t
Convex Exemplar 8 Nodest
SGI Challenge 32 Nodes

^ Ma|ority of funding provided by other Federal agency (ARPA, NIH) or state.
i Donated m full or in part by the manufacturer for extended evaluation

High Performance Computing Infrastructure and Accomplishments

The Centers Prdgram provides a stable sup-
ply of vector computing cycles needed by the
research community while investmg in sig-
nificant capacity of scalable parallel systems,
capable of ultimately growing to a size neces-
sary for full scale grand challenge problems.
While the numbers of Vector Supercom-
puters has decreased, the computing power
represented by that group in fact increased
substantially. However, the increase in the
scalable parallel systems was much larger,
reflecting the growth potential of this type of
computing platform and a strong
NSF/ARPA partnership.

National Access to Vector Multiproc-
essors

The NSF Supercomputer Centers established
in the mid-1980s brought access to state-of-
the-art supercomputers for the first time in at
least 15 years. Indeed, in the 1960s, only a
few universities had such access. This open-
ing of universal access led to an unprece-
dented increase in the number of researchers
and universities involved in advancing the
frontiers of scientific and engineering research
by using high performance computing. By the
early 1990s, some 15,000 researchers in over
200 universities had u.sed one of the Cray
Research vector multiprocessors or the IBM
vector mainframe in one of the NSF centers.
This wide pool of computational researchers
made it possible for the center's program to
begin to respond to the demand for paral-
lelism that had been developed in the Com-
puter Science Community, and adopted by the
most adventurous user. The 90" s saw the
NSF center's program substantially widen its
range of architectural offerings.

Achieving Production Parallelism

The Cornell Theory Center (CTC) became the
first member Center of the NSF MetaCenter
to achieve production parallelism on a vector
supercomputer, with over 1/3 of its vector
supercomputer cycles used for parallelism in
1989. CTC integrated its two (ES/3090 600)
vector supercomputers using a special 200
mbyte/sec hardware interface, allowing paral-
lel jobs the potential of executing across 12

vector processors. Users u.sed a shared-
memory parallel FORTRAN developed by
IBM in a joint project with the Theory Center.

Early Migration to the UNIX Operat-
ing System

During the early and mid-1980's the UNIX
operating system was widely viewed as inap-
propriate for supercomputers for reasons of
performance, system management tools, ap-
plication development and measurement
mechanisms, and security. Cray supercom-
puters were run mostly with operating sys-
tems that were designed at national laborato-
ries (LLNL and LANL in particular) and this
required extensive local software support. In
1987, NCSA became the first major super-
computer center to migrate its Cray super-
computer from CTSS (Cray's proprietary
time-.sharing system) to UNICOS, a UNIX-
based operating system developed at Cray
Research for its supercomputers. This move
to UNIX was the beginning of a merger be-
tween computational science and computer
science, because most computer .science re-
search involved the UNIX operating system at
that time. Coincidentally, CTC was the first
site to run IBM's high performance UNIX
system on its ES/3090 and ES/9000 main-
frames in production.

Early Access to Massively Parallel
Computers

Beginning in 1985, CTC provided ex-
perimental scalable parallel machines, first
the EPS T-series and an iPSC/l parallel sys-
tem, to its user community. In 1988. with the
installation of its iPSC/2 with 32 processors,
CTC made early .scalable computing available
for production use by the national com-
munity. Massively parallel computing was
introduced to the re.search community begin-
ning with NCSA's CM-2 in 1989. Each
Center provided early access to new genera-
tion MPPs. The CTC was the first site to in-
stall an IBM SPI and SP2. PSC installed the
first Cray T3D, and early CMS (NCSA),
Paragon(SDSC), Ncube(SDSC), and Ken-
dall Square (CTC) machines were installed.
Early access to these machines, enabled by

-4-

High Performance Computing Infrastructure and Accomplishments

support from ARPA and NIH, allowed pio-
neering users to explore the benefits of fine-
grained parallelism. Each Center worked
with the user community and the vendors to
develop application codes, which could then
be ported to other platforms. In 1992, the
CM-5 was added to the program at NCSA as
the largest distributed memory parallel super-
computer available to the national academic
and industrial communities. From 1992 to
the present NCSA has worked closely with
national users and the computer .science
community to create a wide range of 512-
way parallel application codes that can in
1995 be moved to other large MPP architec-
tures such as the T3D at PSC. the Intel Para-
gon at SDSC. or the IBM SP-2 at CTC.

Superhnear speedup on heterogeneous
processors

In 1991. PSC was the first site to distribute
code between a massively parallel machine
(TMC-CM2) and a vector supercomputer
(Cray YMP). linked by a high speed channel
(HiPPI). Experiments on applications as di-
verse as molecular dynamics, medical imag-
ing, chemical flowsheeting and gene sequence
alignment showed superhnear speedup (the
applications on the linked system ran more
than twice as fast on each system separately).
This formed part of the motivation for het-
erogeneous computing, as later embodied in
the tightly coupled Cray T3D/C90 systems.
PSC's T3D was the first shipped anywhere.
PSC developed a set of codes for transferring
data between the Cray and CM2 which later
enabled them to communicate between the
T3D and C90 at speeds superior to what was
available from the vendor.

A similar superhnear speedup was obtained
on the CASA gigabit testbed set-up included
parts of two supercomputers (64 nodes of the
528-node Intel Delta svstem at Caltech and
one processor of the CRAY C-90 at SDSC.
150 miles away); two HiPPI-SONET gate-
ways; and a SONET wide-area link between
San Diego and Pasadena, which is itself a
prototype undergoing tests in a collaboration
between MCI and Pacific Bell. Usine an en-

vironment for distributed parallel computing
called Express (a product of ParaSoft Corpo-
ration). Aron Kuppermann and Mark Wu
(Dept. of Chemistry, CalTech) did a test cal-
culation of the reaction of atomic hydrogen
with molecular heavy hydrogen (deuterium)
at a total energy of 2.5 eV. This problem had
taken 100 hours to .solve on the SDSC
CRAY Y-MP a year earlier. On the new C-
90, It took 17 hours. On the Delta alone, it
took 16. But when the problem was distrib-
uted between the C-90 and the Delta, the
whole problem was solved by the two ma-
chines in just under five hours, a factor of 3.3
faster than it could be done on either machine
alone.

Workstation Clusters

Given historical trends showing much more
rapid improvement in microprocessor tech-
nology than in vector technology, many Cen-
ters began exploring workstation clusters, in

1990. NCSA examined the usage of the Cray
Y-MP and determined that a significant
amount of capacity could be gained by mov-
ing appropriate applications to scalar RISC
processors. Based on this study and on pre-
dictions that microprocessor technology
would surpass vector technology by the mid-
1990's. NCSA set up an IBM RS/6000 clus-
ter as a farm of processors used as stand-
alone compute servers. Beginning in 1991,
CTC did pioneering work with IBM on clus-
tered RS/6000 workstations with high-speed,
proprietary communications links, including
an experimental optical switch. The informa-
tion gained during this joint project was used
to guide the design of the SP systems. Also in

1991. PSC set up a cluster of DEC worksta-
tions, and working with Florida State Univer-
sity, significantly enhanced queuing, account-
ing and control software and also integrated
the AFS file system into this environment.
These clusters all served as computing re-
sources and al.so as platforms for the devel-
opment of distributed memory message
passing codes. These projects generated high
interest in industry, and NCSA trained several
dozen industrial sites on the integration, op-

-5'

High Performance Computing Infrastructure and Accomplishments

eration. and management of clusters ot mi-
croprocessors. NCSA is now moving to es-
tablish clusters of shared memory workstation
multiprocessors from Convex/HP and SGI,
while PSC will develop applications for
Cray's new offering, the J90. The CTC was
given supplemental funding by NSF to build
its cluster to 32 processors, providing a com-
patible path to Its (later) IBM SPl and SP2
systems. The system, including experimental
high-speed switch, was used for production
work by the community, which has now mi-
grated to the SPl and SP2 environments.

Portable Parallel Programming
Tools

Although the architectures of massively par-
allel systems differ greatly, the major time
and money investment of the research com-
munity (as contra-sted to the center's person-
nel) is in developing and converting codes
(porting) to operate in different envi-
ronments. The close cooperation between the
NSF Centers via the MetaCenter and infor-
mal contacts among its research users and
cooperating agencies such as ARPA and the
various National Labs have resulted in sub-
stantial progress ensuring that labor intensive
programming operations need not be dupli-
cated needlessly.

Prototype Parallel Programming En-
vironments

Working with the Parascope Group at the
Center for Research in Parallel Computing, an
NSF Science and Technology Center, CTC
developed extensions supporting new parallel
programming paradigms and extensions
making ports from one type of parallel pro-
gramming platform to another easier. ParaS-
cope is a prototype parallel programming en-
vironment. Both the BIMA and Cosmology
GCs at NCSA are working closely with Indi-
ana University Computer Scientist Dennis
Gannon to move application codes previously
written in FORTRAN to the portable pC-n-
which IS the model for HPC-t-i-, the equivalent
of HPF in the C-i-i- world.

Extensions of PVM

In the Dome project, Adam Beguelin, one of
the original developers of PVM now working
jointly at PSC and in Computer Science at
Carnegie Mellon University, is extending
PVM to improve load balancing and fault
tolerance. His work is guided by the experi-
ence of PSC's cluster users.

Scalable Libraries

The goal of the ARPA funded Scalable Paral-
lel Libraries project is to develop mathematical
software libraries for massively parallel proc-
es.sors that are roughly comparable in scope to
the math libraries typically available on con-
ventional supercomputers. Michael Heath and
his group at NCSA are one team in this multi-
institutional project and they have been devel-
oping parallel direct methods for solving
sparse linear systems. For this purpose, they
have developed a fully parallel sparse solver
for distributed memory parallel computers.
Unlike most other efforts, which have focused
only on factorization, this solver performs all
phases of the computation in parallel mode,
including the symbolic preprocessing neces-
sary to reorder the sparse matrix and distribute
it across processors to maintain data locality.
With funding from IBM, CTC staff devel-
oped scalable versions of key numerical li-
brary routines for its IBM cluster system:
these routines were included by IBM in its
ESSL library product.

Storage Technologies

With the vast increase in both simulation and
observational data, the MetaCenter has
worked a great deal on problems of storage
technologies. Here again, many of the biggest
areas of progress are in software. The crea-
tion of a universal file format standard, a na-
tional file system with a single name space,
and a multivendor archiving software are
some of the results of MetaCenter innova-
tion, collaboration with computer scientists,
and with other leading national laboratories.
There are even examples of the Program's
computational facilities being used to im-

-6-

High Performance Computing Infrastructure and Accomplishments

prove the basic storage capacity of the physi-
cal medium ot storage itself.

AFS-Estabiishing a National File Sys-
tem

PSC recognized that the Andrew File System
(AFS). developed at Carnegie Mellon Uni-
versity with IBM support for a workstation
environment, was particularly-well suited for
use in high performance computing, because
of its superior security, .scaling properties,
and manageability. PSC undertook a major
program of adapting AFS to the high per-
formance computing environment which has
led to a MetaCenter wide effort to develop a
shared national file system. PSC's AFS en-
hancements to Cray's UNICOS are installed
at a number of advanced computing centers
(SDSC. .NERSC. IPP. LRZ (Germany).
ETH (Switzerland) and the University of
Stuttgart). PSC has also extended AFS to
multi-resident AFS which enables AFS to be
a component of a hierarchical storage system.
permitting transparent and cost-effective stor-
age of large amounts of data. These en-
hancement are installed at other MetaCenter
sites. NERSC. Cray Research. Transarc Cor-
poration. Ma.x Planck Institute for Plasma
Physics, and the University of Cologne. With
AFS. distributed applications across the
MetaCenter are now possible. For instance.
Paul Dawson (Dept. of Mech. and Aerospace
Engineering. Cornell Univ.). simulating de-
formations of aluminum, uses AFS to build
a distributed application with modeling per-
formed at PSC and visualization at the CTC.
The SDSC is running an AFS cell supporting
the Computational Center for Macromolecu-
lar Structure (CCMS). an SDSC. UCSD. and
Scripps Research Institute collaboration. The
charier of the CCMS is the development and
distribution of portable, innovative sotfware
for the study of macromolecular structure.
AFS simplifies the cross-institution distribu-
tion and maintenance of software and textual
information.

CTC was the first Center to put AFS in pro-
duction on its HPC systems; in fact, all CTC
production systems, except KSR. and servers

are integrated through AFS. including its new
mass storage environment. CTC has lead in a
number of areas of file system integration,
including a project with the University of
Michigan integrating its AFS mainframe
port. IFS. into its HPC environment, and
joint efforts with TRANSARC in improving
AFS performance over high speed networks,
including an FDDI testbed. New efforts in-
volving TRANSARC include optimization
over ATM and within IBM's SP2 scalable
switch.

HDF-Creating a Standard File For-
mat

The NCSA Hierarchical Data Format (HDF).
created by Michael Folk and his group at
NCSA has become one of the leading self-
describing file formats in the world today.
Many scientific institutions, organizations and
programs have adopted HDF as a standard
file format for data exchange and/or archiving.
In 1992. NASA selected ^HDF as the basis
from which to develop an EOSDIS standard
data format (SDF). The goal of SDF is to
provide a single, self-describing format for
distributing data derived from approximately
1.2 terabytes of data daily that EOSDIS will
eventually produce. Other examples of HDF
adoption include the Institute of Applied
GeoScience (seismic data). Pacific Northwest
Laboratory (cancer research). Children's Hos-
pital in Boston (x-ray images), the UCLA
Scientific Visualization Lab. By working
closely with many different user communities
to support the harmonization of data models
and metadata conventions across as many
disciplines as possible. NCSA is helping to
create a software foundation for the Nil en-
abling It to reach its potential to support the
broadest constituency possible.

Migrating to a Standard Archiving
Software

In 1985. the NSF Centers' goal was simply
to establish national access for the academic
community to the type of advanced super-
computing and archiving systems found in
the Dept. of Energy national laboratories.
NCSA and SDSC "duplicated the Los Ala-

High Performance Computing Infrastructure and Accomplishments

mos computing environment in 1985, includ-
ing the Common File System (CFS) archive,
while PSC adopted Westinghouse's PDM
software that had additional data migration
facilities. As time went on, the Centers began
to develop innovation in storage software. It
is the strategy of the MetaCenter to explore
alternative technical approaches to storage at
the bits level, while maintaining interop-
erability through standard protocols. SDSC.
through the DISCOS (Distributed Computer
Solutions) division of General Atomics
(which is now owned by Open Vision), pio-
neered productization of distributed, hierar-
chical file and storage management ap-
plications software for networked, multi-
vendor and open sy.stems environments,
based on the IEEE storage model. Working
with the UIUC Computer Science De-
partment, NCSA was able to encode data
migration and caching strategies into CFS to
improve its ability to minimize disk cache
misses. In 1992, NCSA developed CFS-to-
UniTree data formatting and migration tools
as well as a suite of archive management
tools. PSC integrated Cray's proven com-
mercial archiving technology (DMF) with
more usable front-end software and with its
multi-resident AFS. SDSC collaborates with
OpenVision and the National Storage Labo-
ratory in the further development and stabili-
zation of UniTree as a robust production ar-
chival storage system. SDSC has developed
transaction journaling software that is criti-
cally important for guaranteeing integrity of
the file name.server. CTC and SDSC are in-
.stalling the first generation of a high-
performance variant of UNITREE which has
been developed by the National Storage
Laboratory, to provide enhanced L/O capabil-
ity to balance the increa.se in in.stalled com-
puting capacity. CTC will continue to work
closeFy with IBM in the testing and deploy-
ment of the next generation of mass storage
systems, the High Performance Storage
System (HPSSl. This will include the ca-
pability of utilizing parallel VO to speed data
transfer to the SP2.

Development of high-density magnetic
media

The Center for Magnetic Recording Research
(CMRR), located on the campus of UCSD.
has funding from NSF and 21 corporations
having enterprises connected with the storage
and retrieval of magnetically written infor-
mation. The chief technical problem CMRR
has been addressing has been magnetic noise
in the metallic thin-film media used to coat
high-density disks. Neal Bertram (Depi. of
Electrical and Computer Engineering,
UCSD) is a researcher at CMRR who tackles
the problem of magnetic noise computa-
tionally. Magnetic thin films are poly-
cry.stalline. rather than continuous or amor-
phous, and the grainy, particulate nature of
the medium is a fundamental source of noise.
Bertram's calculations have explored the ef-
fects of two primary types of interaction be-
tween grains that cause noise: magnetostatic
coupling and exchange coupling. The calcu-
lations resulted in recommendations for al-
loys and fabrication processes that would re-
duce noise from both sources. The research
enabled engineers at IBM's Almaden Re-
search Center to design a disk coating that
packs a gigabit (one billion bits) of in-
formation onto a square inch, which is 15-30
times current storage densities. Bertram has
now turned his attention to calculations ot
other effects, including giant magnetoresis-
tance, that can be important in designing
high-density disk media and disk recording
and playback heads. His codes model the
process of recording bits in intricate detail;
the process of laying down a single bit of in-
formation takes several minutes to calculate
on SDSC's Cray Research C90.

Networking

One of the great succes.ses of the NSF Meta-
Center has been in providing the "high-end
pull" that has led to the creation and expo-
nential evolution of the NSFnet. As a result,
the NSFnet backbone of 1995 has 3000
times the bandwidth of the backbone of
1986. The Centers have also prototyped the
high performance local area networks that are

High Performance Computing Infrastructure and Accomplishments

needed to feed into the national backbone as
well as the next generation of gigabit back-
bones. Security over networks is essential not
only for industrial usage, but more and more
for widespread citizen usage. Again, the
MetaCenter has created innovations such as
dealing with mdustrial firewalls. The exis-
tence of the MetaCenter network testbeds al-
lows for new kinds of science to be attacked,
perhaps best illustrated by the rise of telemi-
croscop), m which leading edge projects are
being carried out at each of the NSF Super-
computer Centers. With the national priority
on the .Ml, the Centers are moving rapidly to
expand their networking research to commu-
nity based Nil lestbeds. including local
healthcare, education, government, and small
business partners.

Evolution of NSFNET

The 56kbps connection between the NSF
Centers, established in 1986, was the begin-
ning of the NSFnet. Based on the successes of
ARPAnet and the TCP/IP protocol within the
computer science and Dept. of Defense com-
munities, the NSFnet rapidly grew to provide
remote access to the NSF Supercomputer
Centers by the creation of regional and cam-
pus connections to the backbone. Although
started by the pull from the high end, the
NSFnet soon began to provide ubiquitous
connectivity to the academic research commu-
nity for electronic mail, file transport, and
remote login, as well as supercomputer con-
nectivity. These daily uses soon became in-
di.spensable to the research community and
the sustained exponential growth of the In-
temet took off. The MetaCenter" s industrial
partner network was among the first in the
coiporate world to use NSFnet/lnternet tech-
nology to connect corporations to the Internet
for the purposes of computational science.
This was an important precursor for today's
rapid commercialization of the Internet.

By early 1995, the NSFnet will return to a
high speed backbone connecting the Meta-
Center and some of the newly selected Meta-
Center Regional Alliance members. However,
the bandwidth of the backbone will be 3000

times higher than that of the original backbone
(56 kbps). While .some tend to think of the
MetaCenter as focusing on high performance
computing only, it is useful to remember that
computing power of the fastest supercom-
puter /?n«c'.v.sr.<;- in the program has grown by
little more than 100 times during the same
period. Indeed, it is likely that the 155 mbps
vBNS will be upgraded to 622 mbps nirliin
two years. Even by the time the Centers re-
ceive the first teraflop machines in 1997-98,
realizing a factor of 1000 increase in speed
over 1985. the backbone will have grown by a
factor of at least 25,000 fold in bandwidth.

As part of an SDSC/UCSD collaboration,
Kimberly Claffy recently completed a Ph.D.
dissertation that outlined a methodology for
profiling Internet traffic flows at a variety of
granularities. The methodologies and models
developed as part of this traffic characteriza-
tion effort should prove very useful as the
Internet evolves to an even larger system in
which the traffic composition needs to be un-
derstood, particularly for planning future
technology and capacity.

High Performance LANs

The center's program has also pioneered sev-
eral transitions in local area and metropolitan
area networks both on site and on university
campuses, acting as a prototyping facility for
other campuses who needed to know how to
develop long range networking plans for their
campuses. In 19S8, NCSA in.stalled the first
Ultranet Gigabit LAN networks with multiple
supercomputers and demonstrated 480 Mbil/s
between the CRAY-2 and Cray Y-MP su-
percomputers. In 1989, NCSA replaced tradi-
tional HYPERchannel backbone networks
with the then-emerging 100 Mbit/s FDDI
standard. In 1991. PSC began its move to a
HIPPI-based interconnect between its major
systems.

In 1993, NCSA, several industrial partners,
and the UIUC Computer Science Department
established a local area ATM network testbed
to help corporations gain hands-on experience
with ATM switches and interfaces. Insight
from this ATM testbed has already been used

High Performance Computing Infrastructure and Accomplishments

to develop long-range corporate network
strategies for J. P. Morgan, Phillips Petroleum.
FMC Corporation, and United Technologies.

CTC was the first site to integrate ATM into a
parallel supercomputer environment on its
IBM SPl in April 1994 and its IBM SP2 in
July. ATM will be u.sed for AFS-based file
service and other high speed transport needs,
including distributed applications and image
transport.

Gigabit Testbeds

Since 1987. NCSA and the University of Illi-
nois Computer Science Department have
worked with AT&T on the XUNET research
network testbed with capacity that is one step
beyond what is available on the Internet.
While the NSFnet has moved from 56 Kbs
through 1.5 Mbps to 45 Mbs, XUNET has
moved from 1.5 Mbs through 45 Mbs to 622
Mbs. In July 1993, XUNET was upgraded to
622 Mbs. the first network testbed to intercon-
nect ATM switches using 622 Mbs transmis-
sion technology over long (>50 miles) dis-
tance using pure optical fibers with in-line
optical amplifiers.

PSC has worked with CMU's Computer Sci-
ence Department in the Nectar Metropolitan
area gigabit testbed to develop new network-
ing technology for very high-speed, low-
latency multi-machine interconnects and to
develop the applications base which can bene-
fit from such technology. This work is fully
collaborative with PSC's ground-breaking
systems and applications level work in het-
erogeneous systems. As a result of testbed
work, numerous applications can now run
routinely between advanced machines at
PSC's main hardware facility and tho.se on the
CMU campus, 15 miles distant, at speeds of
up to 1 Gbit/s.

In partnership with NYNEX, Syracu.se and
Rome Laboratory, now extended to Colum-
bia, SUNY Stonybrook and Polytechnic In-
stitute. CTC participated in building a produc-
tion-level ATM network focused on demon-
strating research and commercial applications.
This network was demonstrated to the Gover-

nor of New York in January 1994. NYNET
is also a Nil te.stbed. involving outreach,
medical applications, video on demand, as
described in a later section.

Various applications are being tested in the
CASA te.stbed. in which SDSC is a major
partner. Besides the chemical reaction dy-
namics, led by Aron Kuppermann of Caltech
and mentioned above in the section on super-
linear speedup, there is a coupled atmo-
sphere/ocean model developed by the group
led by Roberto Mechoso at UCLA. Another
is Calcrust, a project directed by JPL. which
has u.sed distributed heterogeneous comput-
ing over the CASA links to combine satellite
imaging, .seismic data, and surface topogra-
phy in visualizing the foci of aftershocks of
the 1992 Landers earthquake in Southern
California.

Secure Networks

In 1987 NCSA installed a 1.5 Mbit/s DS-I
connection to Eastman Kodak in Rochester,
New York, followed by another DS-1 con-
nection to Amoco laboratories in Chicago and
Tulsa. By 1990 NCSA had connected over a
dozen indu.strial laboratories to the Internet us-
ing a combination of innovative security pre-
cautions. The.se included various forms of
"firewalls" which have now become com-
monplace on the Internet.

New Science Enabled by Networks—
Telemicroscopy

The San Diego Microscopy and Imaging Re-
source (SDMIR), led by UCSD neurocy-
tologist Mark Ellisman. is an NIH-funded
Research Resource centered on a new, fully
computerized Intermediate Voltage Electron
Micro.scope (IVEM). The IVEM is used to
look at comparatively thick tissue sections (2-
10 microns) and it has been employed in
studies of cortical neurons with and without
symptoms of Alzheimer's, in .studies of an-
other type of brain cell, called a Purkinje neu-
ron, and studies of cell membranes. A long-
term collaboration between SDMIR and
SDSC has made the microscope usable inter-

10-

High Performance Computing Infrastructure and Accomplishments

actively, over the Internet, coupled to the
SDSC Cray Research C-90.

Computational analysis and simulation is al-
lowing biomedical researchers to study and
predict the activity of potential new drugs at
the molecular level. CTC is working jointly
v\ ith Steven Ealick. et al.. director of Mac-
CHESS. a group using the Cornell High En-
ergy Synchrotron Source for Macromolecular
Modeling. Using existing high-speed connec-
tions between CTC and MacCHESS. the
pioieci IS building the capability, for the first
time, for pharmaceutical companies and aca-
demic researchers to interact dynamically with
x-ray crystallographic analyses at the syn-
chrotron, rather than discovering long after the
beam run that the sample was defective or the
beam positioning non-optimal. While initially
the researchers are using processors on the
CTC SP systems, ultimat'ely a small IBM SP
may be installed at the synchrotron site used
for dynamic analysis, with longer-scale simu-
lation needs being met using the far larger
SP2 at the CTC.

PSC is working with the Center for Light
Microscopy and Biotechnology, an NSF Sci-
ence and Technology Center at Carnegie
Mellon University, to develop an Automated
Interactive Microscope. This microscope will
couple leading edge-microscopy and high per-
formance computing through high speed net-
works allowing the real-time tagging of
chemical reactants in the cell. It will open new
research horizons in biology by giving re-
searchers the ability to control the release of
chemically active agents at critical moments in
cell life, and to monitor the celFs subsequent
development.

The personal computer controlling a scanning
tunneling micro.scope (STM) in the Beckman
Institute at UIUC used software integrated
over a LAN with the NCSA Convex C3880.
TMC CM-5. and SGI graphics workstation to
enable realtime imaging and nanolithography
of silicon surfaces in order to create novel
quantum electronic devices. Working with the
laboratory's director, Joseph Lyding (Dept. of
Electrical and Computing Engineering, UIUC

and Beckman Institute) and iiuenioi of a
widely used STM, NCSA staff members
Rachael Brady and Clint Potter extended this
lelemicroscope to the Internet and demon-
strated the feasibility of using advanced im-
aging instrumentation linked with advanced
computing capabilities from anywhere in the
world. The project was featured in the special
issue of Research and Development Mai^a-
:ine (Oct 25. 1993) on Winning in the 21st
century.

Nil Testbeds

As a partner in Common Knowledge; Pitts-
burgh, an innovative project introducing net-
working and computing into the entire Pitts-
burgh School District. PSC is working with
numerous partners, including Digital Equip-
ment, Apple Computer and both telephone
(Bell of Pennsylvania) and cable TV (TCI)
companies, to create a prototypical, cost-
effective approach to widespread use of ad-
vanced technology in public education.

In collaboration with researchers and physi-
cians at the University of Pittsburgh Medical
Center, the PSC is developing an Nil-based
digital library of pathology images, and the
applications and software technology which
will enhance the practice, teaching and cost-
effective delivery of pathology.

NCSA in collaboration with UIUC and the
Champaign County Chamber of Commerce
have been building CCnet. an Nil testbed
during the last 18 months. Over 200 people
from over 70 community organizations have
been involved since April, 1993 in defining
six major applications experiments in small
business, health care, education, government
and community services, agribusiness, and
geographic information systems (GIS). The
first of 20 multimegabil/s links into the com-
munity was established with the Urbana Free
Library in August 1994. All the high .schools
and a number of small businesses are hooking
on in September. NCSA is establishing a
large GIS server which will be available over
CCnet to community projects. Partners in
CCnet include Time-Warner cable.

11-

High Performance Computing Infrastructure and Accomplishments

Ameritech. DEC, Motorola, and potentially
MCI and AT&T.

NYNET, one of the gigabit testbeds, is also
designated as an Nil testbed providing out-
reach, medical appHcations, video on demand
to CTC and New York academic and indus-
trial partners.

InterNIC is the latest in an evolutionary line of
support from the NSF for the use of the In-
ternet by the science, research, and education
communities. The InterNIC provides three
types of services: Information Services
(provided through General Atomics and the
SDSC), Directory and Database Services, and
Registration Services. Information Services
provides procedures for connecting to the
Internet, pointers to resources and tools avail-
able over the network, training seminars for
new and experienced users and up-to-date
reports on new resources and activities on the
Internet. Several innovative approaches to
distributed .services have been implemented,
including the InfoGuide, an on-line Intemet
information service. The Scout Report is a
weekly summary of Internet highlights which
combines in one place the highlights of new
resource announcements and other news that
occurred on the Internet during the previous
week. The InterNIC Reference Desk acts as
the "NIC of first and last resort." The desk
supports a variety of users answering
"starter" questions from novice users who are
unfamiliar with the Internet as well as spe-
cialized questions from intermediate and ad-
vanced users.

Visualization and Virtual Reality

The NSF center's were instrumental in
bringing the notion and tools of scientific vi-
.sualization to the research community in the
1980s. By combining advanced visualization
resources with simulation data.sets created by
remote users on the centers program's high
performance computers, new visualization
paradigms for interpreting numerical data
were developed. This led scientists to consider
visualization as an intimate part of their com-
putational toolkit. In addition, the centers
worked closely with the pre-existing computer

graphics community to get them creating new
tools for scientists as well as for entertain-
ment. Already by 1 987. the staffs of the cen-
ters, working with national users, were creat-
ing scientific visualizations so compelling that
they became regularly chosen to be part of the
SIGGRAPH Film and Video Show, the
"academy awards" of the visualization indus-
try. Today the centers visualization staff and
their allied visualization Centers are at the
forefront of research into how to turn virtual
reality technologies into useful tools for scien-
tific and engineering research.

Development of Scientific Visual-
ization

From its inception, NCSA has worked with
computer artists like Donna Cox (UIUC Dept.
of Art) and Dan Sandin (UIC School of Art
and Design) and computer scientists like Tom
DeFanti (Dept. of Electrical Engineering and
Computer Science, UIC) to create cross disci-
plinary teams with end users in order to create
new levels of scientific visualizations. NCSA
also hired a number of staff from leading
companies in the California entertainment
industry to bring the software tools of special
effects in movies or TV commercials to the
use of the scientific and engineering commu-
nities. Initially, the NCSA visualization envi-
ronment was built on Alliant shared-memory
multiprocessors using the Wavefronl visu-
alization software. As specialized graphics
hardware became available on Silicon Graph-
ics systems, the NCSA visualization envi-
ronment migrated from the Alliant to Silicon
Graphics systems. The scientific visualiza-
tions created by NCSA staff have not only
broken new ground for scientists viewing
their data, they have also won awards world-
wide for aesthetic quality. NCSA's Renais-
sance Experimental Laboratory (REL), cre-
ated by Donna Cox with a major donation
from Jim Clark (founder of Silicon Graphics)
was the first advanced visualization training
facility in the centers and continues to support
university courses in geology, mathematics,
graphics design, computer .science, and other
disciplines.

12-

High Performance Computing Infrastructure and Accomplishments

The SDSC has developed a variety of soft-
ware tools that can be used to access and con-
nect existing visualization resources automati-
cally. The goal has been to provide researchers
with training and access to tools that will sup-
port their research needs. The tools include: 1 )
The SDSC Image Library, a collection of im-
age manipulation and conversion utility rou-
tines that can be embedded in existing soft-
ware. An interesting example of the use of
this library are the image conversion modules
developed by the International AVS Center,
which quickly became the top 2 user modules
in their distribution; 2) the SDSC Image
Tools, a collection of utilities based on the
Image Library, are software tools for reading,
writing, and manipulating raster images. This
toolset also allows researchers to convert file
formats among over thirty widely used
graphics formats (e.g., from HDF to
PICT). Now in its .second release, it runs on
Cray Research. DEC. HP. IBM. SGI, and
Sun Micro.systems platforms. Over 4,000
sites worldwide have uploaded the Image
Tools from SDSC's anonymous FTP area; 3)
vpr, a client-server visualization hardcopy
utility, vpr takes advantage of the Internet by
allowing remote users to send images to local
hardcopy devices. Popular hardcopy devices
that are connected to vpr include a variety of
film recording devices, a color paper plotter,
and video recording; and 4) the SDSC Color
Tutorial, a SuperCard-based hypertext explo-
ration in color theory for computer graphics.
Examples show the different points that are
being made, while hyperlinks allow the u.ser
to jump to the most interesting references.

CTC Visualization staff developed Visual
Programming Language for Animation
(VPLA). a program that easily integrates
sound and image .sequences into scientific
animations. VPLA can use rendered images
from several standard visualization packages
including DataExplorer. As the national re-
pository for Data Explorer software and lead
training site, the CTC works with faculty,
industrial users and students across the
country in developing state-of-the art anima-
tions and images. In addition, using DX, the

CTC has spearheaded using visual program-
ming languages not only for visualizing, but
for managing distributed applications running
across the centers program. For example, the
CTC built a Data Explorer module allowing
its researchers to access the CMS at NCSA.
The CTC was instrumental in IBM's agree-
ing to support DX across all major vendor
platforms, including SGI.

PSC has concentrated its visualization efforts
on the development of tools for remote users.
Its GPLOT software is in use at over 300
sites. Its automated animation facility has en-
abled researchers to produce hundreds of
videotapes without physically visiting the any
specific center. It is now turning its efforts to
develop such tools embodying virtual reality.

Virtual Reality Impacts Industrial De-
sign

In 1992 NCSA began a transition from the
now-traditional workstation visualization ac-
tivities to virtual environments, with leader-
ship provided by Caterpillar Inc., an NCSA
Industrial Partner. Traditionally, translating
electronic CAD blueprints into full scale
wooden models of new heavy earth moving
equipment in order to evaluate design
changes required 6-9 months. Working with
NCSA staff. Caterpillar built up a VR labo-
ratory in the UIUC Beckman Institute and
networked the SGI graphics workstations
which create the VR images to their Peoria
headquarters facilities. Using a variety of VR
viewing technologies, a number of design
options already have been tested for new
models of Caterpillar wheel loaders and
backhoe loaders that will be introduced by
1996. Design changes can now be made in
less than one month. Caterpillar design engi-
neers Dave Stevenson and John Bettner re-
ceived the 1993 NCSA Industrial Grand
Challenge Award for their innovative work.
Media coverage of this award reached over
200 media outlets.

-13-

High Performance Computing Infrastructure and Accomplishments

Development of Immersive Science
Projects

The transition from stand-along workstation
visualization to Nil distributed visualization
was emphasized at SIGGRAPH"92 in Chi-
cago when NCSA collaborated with the UI-
Chicago's Electronic Visualization Labora-
tory and dozens of science teams to demon-
strate wide area interactive visualization at
Showcase. A new level of realism in virtual
reality was debuted there as well with the
public showmg of EVL's Cave Automated
Virtual Environment (CAVE), which pro-
vided complete immersion in complex 3-D
data sets at workstation levels of resolution.
In 1993-1994, EVL, NCSA, and Argonne
organized a national call for proposals which
resulted in over 60 EVL/NCSA computer
science and visualization staff and graduate
students helping researchers from over 30
institutions in porting their applications into
the CAVE environment. For the first time,
this included realtime coupling to parallel su-
percomputers so that dynamic 3-D evolu-
tions could be viewed immersively and
steered interactively. At SIGGRAPH 94,
8,000 attendees were able to directly experi-
ence these science projects. CTC developed a
specific Cave Visualization on Macromolecu-
lar Modeling: the Structure of Acetylcholine
Esterase; this visualization was integral to the
researcher's understanding of the molecule's
activity. This application runs not only on the
CAVE'S SGI workstations, but on CTC's
IBM SP2 system as well.

Virtual Reality over ATM networks

In a project IN 1994 with Rome Laboratory,
demonstrated virtual reality techniques over an
ATM network between CTC SGI computers
and Rome Laboratory. Researching ATM
technologies for real time applications and
demonstrating software tools for application
steering important to molecular modeling,
telemedicine and command and control.

The Sequoia 2000 Visualization Group at
SDSC developed a prototype data visualiza-
tion system "Tecate" using virtual reality
technology to address many of the issues in-

volved in exploring the informational content
of networked data .servers. Tecate enables the
browsing for data that resides in repositories
managed by a database management system
via user-interaction with graphical renditions
of objects that represent data features.

Alpha Shapes, Biomolecules, and
Cosmology

Alpha shapes, a form of geometric modeling
developed by the 1993 Waterman Award
winner Herbert Edelsbruner (Dept. of Com-
puter Science, UIUC) and NCSA staff mem-
ber Ping Fu, focuses on the formal definition,
construction, and measurement of shapes for
any given point set in space. The discrete na-
ture of the alpha shape complex has computa-
tional advantages over any other known
method which can be exploited in computing
surface area and volume of a space filling dia-
gram and in localizing and measuring voids.
The latter is useful in studying water mole-
cules residing inside a protein. NCSA users
have discovered other related applications of
alpha shapes by applying them to such diverse
fields as adaptive grid generation, medical
image analysis, visualizing the structure of
earthquake data, and the large-scale structure
of the universe.

Digital Libraries and Infoservers

The National Information Infrastructure re-
quires many software, computer, and com-
munications resources that were not tradi-
tionally thought to be part of high perform-
ance computing. In particular, knowledge
organization, location, and navigating tools
needed to be developed. The NSF Su-
percomputer Center staffs and their as-
sociated universities have proven to be fertile
ground for developing their new tools. Per-
haps the most spectacular success has been
NCSA Mosaic, which in less than 18 months
has become the Internet knowledge browser
of choice by over a million users. The Mo-
saicAVorld Wide Web infrastructure has set
off an exponential growth in the number of
decentralized authoring of information serv-
ers.

-14.

High Performance Computing Infrastructure and Accomplishments

Digital Libraries

In 1989, NCSA as part of its XUNET appli-
cation testbed proposed that multimedia
digital libraries would require gigabit net-
works in order to fully support high defini-
tion imagery and the coupling of large data
sets with computing resources and geo-
graphically dispersed researchers. This re-
sulted in research developments like DICE
(Distributed Collaboration Environment).
Parallel efforts in providing researchers with
global infonnation retrieval and display capa-
bilities over existing environments combined
collaboration with Bruce Schatz (then at U.
Arizona) and his Worm Community Sys-
tem, with internet-based designs of compo-
nent client/server architectures, like the World
Wide Web. These approaches influenced the
design of the current grand challenge digital
library prototype for accessing radio astron-
omy images and data sets. NCSA built on
the success of Internet access tools such as
NCSA Telnet, adapted this modified digital
library paradigm to the Internet with NCSA
Mosaic. Today, the grand challenge image
library uses NCSA Mosaic as it's user inter-
face, and Schatz has joined NCSA to head
the recently awarded NSF/ARPA Digital Li-
braries project, which combines a testbed
based on the component architecture with
experiments in object-based designs.

Scalable Information Servers

The enormous success of NCSA Mosaic and
CERN's WorldWideWeb has resulted in
explosive growth in the use of NCSA's
WWW server. By the end of 1993, NCSA's
server load had grown beyond the capabilities
of any single server. This resulted in the de-
sign of an innovative distributed scalable
.server architecture that involved a modifica-
tion of the Internet's Domain Name System
software. By Sept. 1994, the NCSA WWW
server was handling over 2 million connec-
tions per week. NCSA's Hewlett-Packard
workstation cluster based distributed infor-
mation server has now been duplicated at
many WWW and PTP sites on the Internet
and within coiporations. A number of corpo-

rations are presently working with NCSA on
the next generation of this distributed ar-
chitecture.

The Rise of the MosaicAVWW In-
formation Infrastructure

NCSA developed the Mosaic user interface
software which provides point-and-click ac-
cess to the diverse information storage proto-
cols of the Internet, such as World Wide Web
(WWW), Gopher, FTP, and WAIS. NCSA
Mosaic establishes the necessary connections,
file transmissions, decompression, launch of
viewer programs, and screen display of text,
images, animations, or audio, in response to a
single mouse click from the u.ser. NCSA Mo-
saic is available for Mac, Windows, and Unix
computers for free to individual users, for
government and educational use, and for in-
ternal use within companies. Monthly down-
load rates from the NCSA site alone are con-
sistently over 30,000. Although accurate esti-
mates are difficult, it is widely felt that over a
million copies of NCSA Mosaic are in use.
Further, commercial versions of NCSA Mo-
saic are available. The principal licensee. Spy-
glass, Inc., has announced orders for over five
million copies of their enhanced version, with
projections to twenty million copies within a
year. Use of NCSA Mosaic has increased
WWW traffic on the NSF backbone by over
10,000 fold since Jan. 1993. Overall WWW
traffic in August hit 1.3 Terabytes or 8 % of
the total NSFnet backbone traffic, higher than
SMTP. Because of this, NCSA has become
the second biggest Internet site in the world in
terms of traffic from its site. The NCSA Mo-
saicAVWW information infrastructure is al-
lowing for an enormous growth in decentral-
ized authoring of infoservers throughout the
world. In 1994, NCSA was given Infoworld's
Publisher's Industry Achievement Award.

DESKTOP Software

From the beginning, the NSF Supercomputer
Centers provided focal points for pulling to-
gether teams of computer scientists and
software developers. Since the history of the
centers has greatly overlapped with the
worldwide rise of the personal computer and

15-

High Performance Computing Infrastructure and Accomplishments

workstation, it is not surprising that the soft-
ware developers focused on creating easy-to-
use software tools for the desktop machines
themselves. These tools have had a major
influence on the usefulness of the supercom-
puter facilities to the remote science and en-
gineering community. The collaboration tools
will have a great impact on tying together the
newly emerging electronic teams of .scientists
made possible by the growth of the Internet.

Connectivity Tools

NCSA Telnet was a ground-breaking desktop
application that provided access to the emerg-
ing NSF Supercomputing Centers in the late
■80s. Developed by the new Workstation
Tools Group (later the SDG) at NCSA, this
brought full TCP connectivity to researchers
using IBM and Macintosh sy.stems, signifi-
cantly broadening the participation base be-
yond Unix users, thereby introducing thou-
sands to both the internet and the NSF Centers
Program. Continuously supported up to the
present time, these tools have also led to a
spin-off company Intercon, headed by one of
NCSA Telnet developers.

Collaboration Tools

NCSA has supported a program of research
and development on collaboration technology
for science and engineering researchers for
over 3 years. NCSA Collage, a tool that runs
across MSWindows, Mac, and XWindows
sy.stems, provides the capability to carry on
remote digital conferencing sessions between
researchers. The first live MetaCenter collabo-
rative session using NCSA Collage was held
in 1992 . Collage combines many of the fea-
tures of NCSA's communications and
graphic data analysis tools. NCSA also con-
tinues to innovate in asynchronous col-
laboration tools such as asynchronous col-
laboration tools, hence the interest in annota-
tion and workgroup support capabilities in
NCSA Mosaic. Current work focuses on
choosing and combining the best of these syn-
chronous and asynchronous capabilities in
usable next-generation global collaboration
tools for the scientific and educational com-
munities.

Cornell University's CUSEEME video tele-
conferencing .software, aimed at providing
video teleconferencing on low-end worksta-
tions, is in use at the NSF Centers and as part
of NYNET, NYSERNET and other organi-
zations for routine use, including medical
projects between the CTC and the Cornell
University Medical College in NYC. This
work, funded by NSF and Cornell itself, is
freely distributed and runs on Mac and PC
platforms using inexpensive video equip-
ment. Several national and international col-
laborations have successfully utilized this
software. The centers have gained experience
with traditional video teleconferencing sys-
tems, through its NSF-funded system. It is
now looking at packet video systems using
the vBNS and other network facilities. These
systems will have the capability of moving
the videoconference from a set of specially
equipped rooms to the desktop. An investi-
gation is now underway to develop the opti-
mal system for CTC's requirements.

Graphics Tools

NCSA Image was the first scientific vi-
sualization tool developed for the desktop
viewing of supercomputing output in the pro-
gram. It provided the research community
Mac and Unix based visualization methods
for analysis of huge data sets, as well as cre-
ating some of the first client/server tools
which integrated remote desktop workstations
and personal computers with the center pro-
grams high performance engines.

The SDSC Image Tools are software tools for
reading, writing, and manipulating raster im-
ages. This toolset also allows researchers to
convert file formats among over thirty widely
u.sed graphics formats (e.g., from HDF to
PICT) and includes extensive C library func-
tionality for creating custom image-
manipulation applications. Now in its second
release, it runs on Cray Research, DEC, HP,
IBM, SGI, and Sun Microsy.stems platforms.
Binaries and sample source code are available
in the public doinain by accessing SDSC's
anonymous ftp area (ftp.sdsc.edu).

16-

High Performance Computing Infrastructure and Accomplishments

Scientist's Workbench

The Scientist's Workbench is an X and Motif-
based software package developed at the CTC.
The main functions of the Scientist's Work-
bench are to bring together the tools and soft-
ware required by scientific researchers in a
distributed computing environment, to pro-
vide a graphical interface to access those tools.

and to provide the software necessary to allow
researchers to easily build their own graphical
interfaces. This tool has been used at most of
the CTC's Smart Nodes (affiliates) by users
and as part of teaching environments for high
performance computing, as well as at the
other centers and by companies and national
labs developing "custom" programming in-
terfaces for their communities.

Accomplishments in Education and Outreach

Education

Familiarity with the tools of computation and
visualization is quickly becoming a sine qua
lion for both researchers and the public. The
spread of access to these tools, like access to
the telephone and television before them, is a
democratizing force in itself: the world of the
shut-in is opened up, the disadvantages of
distance are minimized, the exchange of
techniques and knowledge is enhanced. Edu-
cation, training, and outreach are thus fun-
damental to the programs of the MetaCenter.
Each member of the centers program has
developed educational programs targeted to a
variety of constituencies: university research-
ers, graduate students, undergraduates, edu-
cators at all levels, and K-12 students.

Researchers and Students

One- or two-day workshops are offered by
centers program staff to researchers on site
and at associated institutions, covering intro-
ductions to the computational environments,
scientific visualization, and the optimization
and parallelization of scientific code. In ad-
dition, special workshops have been offered
throughout the centers program on the use
and extension of computational and visuali-
zation techniques specific to various disci-
plines (from biochemistry to lattice gauge
theory). On the campuses of centers program
institutions and on other campuses, centers
program scientists and engineers are active
teachers, either through regular academic ap-

pointments or as adjuncts, lecturers, seminar
leaders, or teachers in extension divisions.

Graduate students often receive fellowships
or similar appointments at centers program
institutions, as their contributions may benefit
a large academic research community or the
computational community generally. As an
example, Ph.D. student Kimberly Claffy
(Computer Science, UCSD) recently com-
pleted her dissertation on a flow-based meas-
ure of Internet traffic that she developed as a
Junior Fellow at SDSC. Dr. Claffy's tech-
nique is the first to permit traffic characteri-
zation on the basis of a temporally and spa-
tially flexible unit, and it is thus an enabling
technology for further advanced network re-
search at SDSC and elsewhere.

The centers program has contributed to the
research projects of hundreds of graduate
students through stipends, access to re-
sources, and relations with centers program
researchers. Each Center fosters collaborative
research by multidisciplinary, multi-
institutional teams of computer scientists,
research scientists, and engineers; postdoc-
toral research associates; and graduate stu-
dents from the national and international
community. These teams forge new ap-
proaches to previously insoluble research
problems, develop community codes, and
host workshops and seminars to transfer
technology.

-17-

High Performance Computing Infrastructure and Accomplishments

Supercomputer Centers Educational Activity Support Summary

Educational Activities

FY91

FY92

FY93

High School

Institutes

Attendees

128

131

121

Other K- 12 Events

Attendees

715

1 .370

1 .985

Research Institutes

1 1

Attendees

262

377

390

Training CoursesAVorkshops |

On Site Events

102

134

Attendees

1.414

1.773

1.929

Off Site Events

Attendees

295

622

104

Seminars/Colloquia

Events

138

114

132

Attendees

2.251

2.788

3.085

Academic Course Accounts

Monthly Newsletter Circulation

234.986

247.692

165.176

Visitors

13.506

16.380

16.392

For undergraduates, the Research Ex-
periences for Undergraduates programs,
funded by NSF, bring in undergraduates to
work for a summer or a school semester or
quarter on specific projects devised by cen-
ters program researchers and/or faculty advi-
sors. The projects are significant in their
scope of computational science and in many
instances have resulted in presentations at
meetmgs and publications. A special project
is CTC's Supercomputing Programs for Un-
dergraduate Research (SPUR), in which stu-
dents apply to work on one of a selection of
projects developed by Cornell faculty in col-
laboration with CTC. One REU student at
SDSC went on to win the top prize in the
Westinghouse Science Talent Search in 1991.
Another developed a program to teach the
use of the Braillewriter to blind students,
which was presented to the Commission on
Equal Opportunity in Science and En-
gineering at NSF (this student, herself blind,
is now a successful computer scientist in
Silicon Valley). Undergraduate assistantships

and internships are also available in the cen-
ters program. Undergraduate student pro-
grammers have worked on many research
problems including numerical weather pre-
diction, the visualization of numerical
spacetimes, and social network analysis.
They have developed numerous applications
and utilities to improve the computational en-
vironment for MetaCenter researchers. Stu-
dents have also worked on library, visualiza-
tion, and educational projects. The REU pro-
grams have been ongoing in various forms
for more than five years.

Outreach to Educators

One particularly effective approach to edu-
cating the next computational generation is
the training of teachers, and many centers
program efforts have been devoted to teacher
training and curriculum development.

Common Knowledge: Pittsburgh is a na-
tional pilot program developed by PSC, the
University of Pittsburgh, and the Pittsburgh
Public Schools to institutionalize educational

18-

High Performance Computing Infrastructure and Accomplishments

technologies within the Pittsburgh PubHc
School District by having PSC implement
the network infrastructure and develop spe-
cific curriculum-based network and computer
applications. PSC's High School Initiative
(1992-1994) involves studentyteacher teams
using PSC facilities to develop computational
tools for inclusion in their schools' science or
mathematics curriculum, with an emphasis
on integrating high-performance computing
into the curriculum and thus bridging the gap
between textbook instruction and real world
applications of science.

SuperQuest is a program involving centers
program sites that brings teams of teachers
and students from selected high schools to
summer institutes to develop computational
and visualization projects that they work on
throughout the following year. In addition to
educational workshop programs associated
with SuperQuest. NCSA has developed five
interactive simulation programs now being
tested in classrooms across the country and
around the world. These include GalaxSee,
an N-body simulator of galaxy formation and
interaction; the Fractal Microscope, which
enables the exploration of self-similar pat-
terns; SimSurface and SimElevator, simu-
lated annealing programs; and LaplaceSeein',
an electrostatic potential solver. Students can
change initial conditions and watch the
simulation evolve as the parameter space is
explored.

SDSC's computational .sciences curriculum
coordinator, Kris Stewart (who is a professor
of mathematics at San Diego State Univer-
sity) has conducted summer workshops,
funded by NSF and Cray Research, with fac-
ulty from primarily undergraduate institu-
tions to develop ways of incorporating high-
perfoiTnance computing into the curriculum.
Stewart uses the workshop materials in her
own SDSU classes, Supercomputing for the
Sciences and an Introduction to Compu-
tational Analysis. SDSC is now halfway
through a three-year, NSF-funded Super-
computer Teacher Enhancement Program

targeted to high-school teachers whose
classes contain underrepresented minorities.

Dr. Bruce Land, of the CTC, has developed
an undergraduate course in scientific vi-
•sualization and computer graphics, using the
data flow block diagram capabilities of
IBM's Data Explorer software. This cur-
riculum, lab exercises and the resulting stu-
dent projects have been shared with the larger
educational community through the CTC's
Education and Training home page. Addi-
tionally, Prof. Steve Vavasis has developed
an interdisciplinary course in .scientific com-
puting using high performance computing
for graduate and undergraduate students.

Over the past year, the CTC had established
the Data Explorer repository, a full .set of tu-
torials for parallel computing on diverse plat-
forms, a complete .set of lecture notes for use
by educators as well as researchers, and a
gateway to materials on the network for sec-
ondary school .science and mathematics edu-
cation.

The educational outreach programs of the
centers program enable students to expe-
rience the advantages of connectivity and
training in all aspects of modern compu-
tational practice. The challenge to effectively
deliver centers program resources to all class-
rooms is being met mainly by the distance-
defeating and multiplicative effects of high-
performance computation itself. The dis-
semination of training and curriculum mate-
rials over the National Information Infra-
structure is a major way in which the suc-
cessful pilot programs can be turned into a
new class of educational resources.

Outreach

Because advances in high-performance com-
puting and communications (HPCC) are
driven by the needs of the practitioners with
the most advanced problems, the centers pro-
gram's .scientific mission includes the con-
struction of an extensive web of relationships
with research and development efforts in
American industry and commerce. Collec-
tively, the centers program's outreach pro-

19-

High Performance Computing Infrastructure and Accomplishments

grams represent a long record of sustained
collaboration among scientists, HPCC de-
velopers, and industrial researchers. Another
aspect of outreach is the effort to find and
serve local and regional needs of govern-
ment, schools, and communities. Some as-
pects of these activities are discussed below.
Application of Scientific Computation
and Visualization to Industrial Pro-
duction
Half of the partnerships between the individ-
ual centers and industry are collaborations
with major industrial firms. These include
American Cyanamid, Amoco, Alcoa,
AT&T. Caterpillar, Corning, Dow Chemical,
Eastman Kodak, Eli Lilly, FMC, Gencorp.
General Dynamics. Hughes Aircraft, IBM,
JP Morgan, Martin-Marietta, McDonnell-
Douglas," Merck Research Labs. Motorola.
Parke-Davis. Philips Petroleum. Schlumber-
ger, USX, and Xerox.

In their original form, the partnerships repre-
sented the first introduction of large-scale
computation and visualization into the store
of resources possessed by even the largest ot
these Fortune 500 companies. While the
companies are for the most part fully com-
puterized now, the majority of these partner-
ships continue today because centers pro-
gram expertise has been essential to the in-
troduction of new ways of employing the
resources of supercomputing: the algorithms,
visualization routines, and engineering codes
are being combined in ways that result in
such advances as high-end rapid prototyping
of new products. As a result, for example.
Eli Lilly maintains its partnership although
the company has purchased its own super-
computer — the useful interactions with cen-
ters program scientists, consultants, and
visualizers continue. In many of these ar-
rangements, the industrial partner's re-
searchers are frequent visitors to the NSF
centers, and centers program researchers also
visit the partner's installations.

Thus, while it is extremely important to Al-
coa that it was able to produce an optimum
aluminum can. to Gencorp that it was able to

design a better injection molding process, to
McDonnell-Douglas that it could perform
rapid airfoil analyses, to American Cyanamid
that it could reformulate soil enhancers, the
sum of these long-term relationships is im-
portant in another dimension as well. The-
Center outreach efforts are helping to revital-
ize American industry, making it more com-
petitive in an increasingly competitive world
market.

Impact on Vendors of High Per-
formance Computing Equipment

The centers program has had a major impact
on the vendors of major high performance
computing equipment. All Centers have
taken early prototypes of machines, have
provided national access to the largest scale
version of such machines, and provided
critical feedback to the vendors. Several have
entered into strategic development efforts
with the vendors. For example, PSC is a
formal partner with Cray Research in the de-
velopment of applications for its massively
parallel T3D. Cray has also internalized
some of PSC's file sy.stem developments.
CTC played an integral role in IBM's re-
entry into the High Performance Computing
arena, as the first customer for its IBM
ES/3090 vector supercomputers and as a
partner in the design and development of its
parallel FORTRAN products. CTC, through
Its director Malvin H. Kalos, was a key in-
fluence in IBM's decision in 1991 to build
the IBM SP systems and ensured that IBM
adopted a strategy that was scalable ulti-
mately up to the teraflops and down to the
desktop.

SDSC has established a clo.se collaboration
with the Supercomputer Systems Division
(SSD) of Intel Corporation to develop sys-
tems software to support multi-user systems,
to serve as a test site for new operating sys-
tem releases, and to improve the .stability of
the Paragon system. SDSC staff have de-
veloped MACS, the Multi-user Accounting
and Control Sy.stem, which Intel offers as
part of the Paragon's standard operating sys-
tem software. This system includes a dy-

High Performance Computing Infrastructure and Accomplishments

namic job mix scheduling algorithm, a port
of the Network Queuing System batch job
submission software, and CPU quota and
accounting systems to control resources used
by separate projects.

SDSC has also collaborated with Cray Re-
search to develop support for multiple-user
systems on Cray systems. SDSC staff have
developed a resource management system
that controls access to various resources on
the system (CPU. memory, and disk) and a
dynamic job mix scheduler (DJMS) to dy-
namically adjust the workload for optimal
performance. SDSC has recently run a T3D
emulator on the Cray C90 and is providing
feedback on its performance. SDSC now
plans to install and evaluate Cray's new
FDDI card, a fiber-optic high-speed network
interface.

Digital Equipment Corporation recently
awarded UCLA and SDSC an external re-
search grant to acquire nine Alpha 3000
model 400 workstations. The cluster, which
has a peak speed of 1 .2 Gflops and is con-
nected at 100 Mbps via a Gigaswitch, will be
used primarily for climate studies led by Dr.
Roberto Mechoso of UCLA. It will also be
available for scientific use and performance
testing by the SDSC user community. SDSC
staff are collaborating with DEC to port the
global climate model to the Alpha cluster
using DEC's High Performance Fortran
compiler.

DEC is a major supporter of Project Sequoia
2000, a collaboration of scientists, computer
and information experts, government agen-
cies, and industrial sponsors to develop an in-
formation-management system for studying
global climate change. The Sequoia vi-
sualization group, centered at SDSC, has
been developing a system that will build on
the strengths of existing hardware and soft-
ware to support next-generation visualiza-
tions. Recently, the group collaborated with
Kubota-Pacific Computer Corporation to
combine Kubota's Denali system with DEC
Alpha machines for advanced 3D color
graphics capabilities. Such collaborations

have benefited DEC. Kubota-Pacific. and the
Sequoia 2000 project.

The National Storage Laboratory (NSL). a
consortium that is developing next-generation
high-speed storage devices, has selected the
UniTree system as the production archival
storage system for all centers program sites.
UniTree, which was originally developed at
the Lawrence Livermore National Labora-
tory, was commercialized by DISCOS, a
spin-off of SDSC and a former division of
SDSC's parent company. General Atomics.
DISCOS was. in turn, sold to OpenVision.
which continues to market the product.

SDSC staff are collaborating with NSL.
IBM. and OpenVision to implement the
NSL's base version of the UniTree archival
storage system on the center's IBM RS/6000
model 980 workstation. They are adding a
new. more robust name server developed by
Lawrence Livermore National Laboratory for
OpenVision, transaction journaling (which
allows reconstructing the database in case of
catastrophic failure), and adding and enhanc-
ing system administrator tools.

The integration of PSC's Multi-resident AFS
into the NSL UniTree environment is
planned to provide user-friendly access for
the centers program members into SDSC'S
NSL archival storage system. The future in-
tegration of HIPPI-attached peripherals, in-
cluding high-speed, high-density tape and
high-speed RAID disk arrays, using third
party data transfer is planned as a way to
substantially increase both archival system
storage capacity and data transfer speed.

A prototype HPSS parallel I/O archival stor-
age system is also planned for evaluation as
the follow-on to the NSL UniTree system.
This system will support striping across
multiple high-speed peripherals to even fur-
ther increase the speed of file transfers.

Stimulation of New, Computationally
Dependent Ventures

About a fourth of the industrial partnerships
are with smaller and newer firms, many of
them leaders in biotechnology. Some are

-21

High Performance Computing Infrastructure and Accomplishments

firms designing new pharmaceuticals (e.g.,
Agouron Pharmaceuticals, Genentech), oth-
ers develop and market the software pack-
ages required for these enterprises. Biosym
Technologies, for example, is working with
both CTC and SDSC to develop parallel ver-
sions of its popular Discover and Insight
packages.

Outreach efforts of the centers program have
resulted in actual spinoff ventures as well.
The commercialization of the software devel-
oped at individual centers is being undertaken
by a number of companies. For example,
NCSA Telnet has been commercialized by
Intercon, and Spyglass will relea.se a package
containing upgraded versions of image tools
and Mosaic. Some 20 companies have now
licen.sed NCSA Mosaic. CERFnet, a Califor-
nia wide-area network for Internet access has
pioneered in supplying access to library and
other large databases; and DISCOSAJniTree,
a mass storage system, is in use at more than
twenty major computer sites. A new mo-
lecular modeling system, called Sculpt, de-
veloped at SDSC, is being commercialized
by a new company. Interactive Simulations.
Sculpt enables drag-and-drop molecular
modeling in real time while preserving
minimum-energy constraints; its output was
featured on the cover of Science last May.

Development of Nationally Valuable
Reservoirs of Skill

About a fourth of the partnerships between
the centers and industry are collaborations
with manufacturers of high-performance
computational, networking, telecommunica-
tions, and visualization equipment. Of par-
ticular interest here are the several partner-
ships funded by the NSF and ARPA through
CNRI to construct and test the "gigabit test-
beds," prototypes of the connectivity that will
be required for the future International In-
formation Infrastructure. Both academic and

industrial research groups are developing the
codes to test the connections, even as manu-
facturers develop the connections themselves,
and the experts assembled in the centers pro-
gram supply links in the form of .specifica-
tions and software.

Community Service

Local and regional outreach efforts range
from the tours given at all centers program
installations through the hosting of visits by
national, regional, and local officials and
commissions, to the kinds of full-scale part-
nerships mentioned above. The NCSA rela-
tionship with the Champaign County Cham-
ber of Commerce has resulted in the forma-
tion of a nonprofit public network, CCnet,
which is already benefiting the Chamber it-
self as well as local schools. Plans are in the
works with Time-Warner to start pilot tests
of the use of public-access cable for a data
highway.

SDSC is working with the City of San Diego
on plans to connect all units of city gov-
ernment, including a high-technology re-
source center to be developed with De-
partment of Commerce funding that will
connect local industry (with a lot of defense
reconversion efforts) to business and com-
putational resources, including SDSC itself.

PSC is exploring extension of the technology
it developed for its Common Knowl-
edge;Pittsburgh K-12 project to embrace
major units of city government.

Outreach is also represented by the publica-
tioi programs of the centers program, the
production of scientific videos and/or multi-
media CD-ROMs, and a collaborative pro-
gram for maintaining a lively and informative
presence on World-Wide Web servers,
which will make information on the pro-
grams easily accessible over the NIL

Important Science and Engineering Accomplishments

Summaries of computationally interesting problems in the NSF Centers Program by the
national science and engineering communities:

-22-

High Performance Computing Infrastructure and Accomplishments

Quantum Physics and Materials

The great disparity between nuclear, atomic,
or molecular scales and macroscopic material
scales, implies that vast computing resources
are needed to attempt to predict the char-
acteristics of bulk matter from fundamental
laws of physics. Therefore, it is not surpris-
ing that since the beginning of the NSF Cen-
ters program this area of science has brought
us some of the largest users of supercom-
puters. Materials .scientists have often been
among the first group of researchers to try
out new architectures that promise even
higher computational speeds.

Below are outlined some outstanding exam-
ples of studymg properties of bulk matter
from extreme conditions, such as occur in
nuclear collisions, the early universe, or in the
core of Jupiter; new materials such as nan-
otubes and high temperature super-
conductors; and more practical materials used
today such as magnetic material and glass.

Phase Transition in QCD

The MIMD Lattice Calculations Col-
laboration (MILC) is attacking the Grand
Challenge problem of "the origins of mass."
Their objective is to use the theory of the
forces governing what are called the "strong
interactions" of elementary particles (quarks
and gluons) to calculate the observed masses
and interactions of the particles that are made
out of them: the hadrons, which include the
familiar proton and neutron. The theory is
called quantum chromodynamics (QCD),
and its numerical incarnation is called "lattice
gauge theory," because the quarks and glu-
ons are represented on a four-dimensional
space-time lattice. They have published nu-
merous studies of the mass spectrum of the
hadrons; the transition between ordinary
matter and the quark-gluon plasma, which is
important in the study of the conditions of the
early universe; and the decays of hadrons via
weak interactions. A number of investigators,
coordinated by Robert Sugar (Dept. of
Physics, UCSB), are engaged in this project
including: Claude Bernard (Washington

Univ.), Thomas A. DeGrand (Univ. of Colo-
rado), Carleton DeTar (Univ. of Utah), Ste-
ven Gottlieb and Alexander Krasnitz (Indiana
Univ.). Douglas Toussaint (Univ. of Ari-
zona). Julius Kuti (UCSD). The consortium
has used large allocations of time on a wide
range of MetaCenter computational facilities
including: Intel Paragon (SDSC), TMC CM-
5 (NCSA), clustered IBM RS/6000s under
PVM (CTC. NCSA), Crav Research C-90
(SDSC and PSC)

Phase Transitions of Solid Hydrogen

Calculations by Natalie, Martin and Ceperley
(Dept. of Physics UIUC, NCSA), carried out
on the CRAY Y-MP at NCSA have estab-
lished the series of crystalline phase transi-
tions of hydrogen as it is compressed to sev-
eral million atmospheres of pressure, such as
found in the interior of the giant planets.
Since Wigner and Huntington in 1935
pointed out that a transformation from a mo-
lecular to atomic state is inevitable at high
pressure, there have been extensive specula-
tions on when and how this transformation
would take place. The recent development of
the diamond anvil technique have allowed
experiments to be performed at pressure
slightly lower than the atomic transition.
Those experiments confirmed the existence
of an molecular orientation transition which
had been earlier computationally predicted by
Ceperley at 1.5 Mbar. Extensive and highly
accurate quantum Monte Carlo calculations
on a variety of crystal structures now predict
that the metallic transition will take place
from a distorted molecular hexagonal struc-
ture into an atomic diamond lattice. For this
and other pioneering science, David Ceperley
was awarded the fifth Eugene Feenberg
Memorial Silver Medal in 1994. David's
PhD advi.sor and the third Feenberg awardee
is Mai Kalos. Director of the CTC. himself a
major user of several MetaCenter supercom-
puters.

Prediction of new Nanomaterials

Marvin L. Cohen (NAS) and Steven G.
Louie (Dept. of Physics, UC Berkeley) have

23-

High Performance Computing Infrastructure and Accomplishments

used MetaCenter computational resources
(SDSC. PSC, NCSA Cray Research Y-MP
and C90) to make numerous advances in
computational materials science. Most re-
cently, they have used both first-pnnciples
and tight-bindmg codes to examine the prop-
erties of carbon nanotubes and nanotubes
composed of boron, carbon, and nitrogen.
Carbon nanotube.s-essentially rolled mi-
crosheets of graphite-are well known, thanks
to the work of Sumio lijima and colleagues at
NEC. They have diameters on the order of 1-
20 nm and are producible in the same carbon
arc chambers used to produce fullerenes (also
called "buckyballs"- assemblages of 60 or
more carbon atoms in cage-like structures).
They have interesting capillarity and elec-
tronic properties. Cohen. Louie and their col-
leagues have predicted the structure and
properties of nanotubes made of boron ni-
tride, which appear to be more stable and
controllable in terms of their electronic prop-
erties. They have also predicted nanotubes of
BC_2N. a boron-carbon-nitrogen compound,
whose electronic properties are even more in-
teresting: they should behave like nanoscale
induction coils. Most exciting, the structures
that were predicted computationally are now
being produced experimentally in the lab of
Alex ZettI at Berkeley, where their electronic
properties can be confirmed.

Theory of High Temperature Su-
perconductors

The Nobel Prize in Physics in 1987 was for
the discovery of a new class of high tem-
perature superconductors. Thousands of re-
search papers have been written about these
unique materials, but the battle is .still raging
over the fundamental mechanism that causes
the ^uperconducting transition at Tc ~ 90K
for the cuprate oxides such as YBA2CU3O7.
David Pines (NAS and first Feenberg
Medalist) and Philippe Monthoux
(Department of Physics, UIUC) used the
NCSA Cray Research Y-MP to carry out a
strong coupling (Eliashberg) calculation of
the normal state properties and Tc for the
model experiment-based magnetic interaction

between quasiparticles. They found that when
the full structure of the quasiparticle interac-
tion is taken into account, a superconducting
transition into a d-wave planar pairing siaie
occurs at Tc ~ 90K for comparatively modest
values of the coupling constant. Although
still an area of active research, this computa-
tion lends credibility to the model that it is the
coupling of planar quasiparticles to the ex-
perimentally measured planar electronic spin
fluctuation excitations which determines the
normal state properties (which they show
acts like a nearly antifcrromagnetic Fermi
liquid) and makes possible high temperature
superconductivity.

Magnetic Materials

James Sethna (Laboratory of Atomic and
Solid State Physics. Cornell Univ.) uses CTC
parallel supercomputers to study the dynam-
ics of disorder-driven first-order phase trans-
formations, including 3-D numerical simula-
tions of hysteresis loops. He is developing
scalable parallel algorithms for systems of
size N = 2000^. Sethna's work enables the
prediction of phase transitions with critical
tluctuations, the simulation of orders-of-
magnitude larger systems to explore critical
phenomena, and detailed computational
studies in materials science as applied to
magnetic storage media, metallurgical phase
transformations, and gases adsorbed on sur-
faces.

Ujiderstanding Glass

For ab initio dynamical calculations to be
useful for real materials in an industrial set-
ting, they must be able to deal with ensem-
bles of thou.sands of atoms for dynamical
effects modeled over microseconds. Signifi-
cant algorithmic developments made jointly
at Corning. Inc. and the CTC, coupled with
the much increased capability of the CTC's
IBM SP2 system, allow this threshold to be
crossed for the first time. Postdoctoral Fel-
low Stefan Goedecker, hired jointly by
Corning, Inc. and CTC, has developed new
extremely fast ways of doing tight-binding
which he can parameterize with the ab initio

24-

High Performance Computing Infrastructure and Accomplishments

codes. This is the only approach currently
known that will handle thousands of atoms
for millions of time steps, bringing the re-
searchers close to observing many of the
mysteries involved in glass chemistry which
have been not well understood for over 2000
years.

Biology and Medicine

Living creatures exhibit some of the greatest
complexity found in nature. Therefore, su-
percomputers have made possible unprece-
dented opportunities to explore these com-
plexities ba.sed on the fundamental advances
made in biological research of the last fifty
years. These activities include: inverting the
data from x-ray crystallography experiments
to obtain the molecular structure of macro-
molecules: learning how to use artificial in-
telligence to fold polypeptide chains, deter-
mined from genetic sequencing, into the
three-dimensional proteins; and determining
the function of proteins by studying their dy-
namic properties, as well as how they interact
with each other or with the DNA backbone
from whence they were created.

These insights are beginning to make signifi-
cant impacts on medicine and plant and ani-
mal biology. New fields of computational
science, such as molecular neurosciences, are
being enabled by academic access to Meta-
Center computing and visualization resources
and staff. Corporations are using supercom-
puters and advanced visualization techniques
in collaboration with the NSF MetaCenter to
create new drugs to fight human disea.ses
such as asthma. New insights into economi-
cally valuable bioproducts are being gained,
for instance, by combining molecular and
medical imaging techniques to create "virtual
spiders'" which can be digitally dissected to
understand the production of silk. Finally,
high performance computers are just be-
coming powerful enough that some dedicated
researchers are able to program mathematical
models of realistic organ dynamics, such as
the human heart.

Crystallography

Herbert Hauptman (Medical Foundation of
Buffalo. Inc.) won a Nobel Prize in 1985 for
development of the "direct method" of pro-
tein structure determination from X-ray
crystallographic data. In a collaboration with
Russ Miller (State University of New York
at Buffalo), these researchers have developed
a numerical approach that extends the "direct
method" of determining molecular structure
from X-ray crystallographic data to larger
molecules, beyond its present limit of about
100 atoms. The algorithm they have devel-
oped, called "Shake-and-Bake," runs on a
number of computing platforms (PSC CM-2
and Cray T3D, NCSA and PSC CM-5) and
has proven it.self effective in more than 20
cases at accurately determining the structure
of proteins that have taken as long as 10
years by existing methods, reducing the time
to a matter of hours.

Folding Proteins using Artificial In-
telligence

One of the most pressing problems in mo-
lecular biology is how to determine the fold-
ing and 3-D structure of a protein, given its
sequence. Peter Wolynes (NAS), Zan
Schulten. and coworkers (Dept. of Chemis-
try. UIUC) have developed a novel approach
to this classic problem using elements from
the theory of spin glasses, associative mem-
ory models, and neural networks. Spin glass
theory provides a framework for under-
standing the cooperative nature of the folding
transition and the qualitative nature of the
phase diagram describing the thermo-
dynamics of proteins. Wolynes et al. devel-
oped simulation codes, based on associative
memory Hamiltonians, and characterized
their phase diagrams .semi-quantitatively.
These Hamiltonians are ba.sed on an energy
function which correlates the sequence of the
protein to be folded with those of proteins
whose structure is known. They were intro-
duced several years ago by Wolynes and co-
workers as polymer analogues of the Hop-
field neural nets. Their work, carried out on
NCSA's Cray-2 and CRAY Y-MP, shows

25-

High Performance Computing Infrastructure and Accomplishments

that even primitive associative memory
Hamiitonians can recognize protein structures
from sequences that are only moderately re-
lated to those already existing in the database.
These procedures are somewhat similar in
effectiveness to the rule-based homology
modeling.

Protein Kinase solution

The Computational Center for Macro-
molecular Structure (CCMS). founded in
1990, is an NSF-funded joint project of
UCSD. SDSC. and The Scripps Research
Institute, with collaborators from ail over the
country. The center made headlines in 1991
when a group led by Susan Taylor (Dept. of
Chemistry. UCSB), one of the principal in-
vestigators of CCMS published the three-
dimensional structure of the catalytic unit of
cyclic-AMP-dependent protein kinase, or
cAPK. This was the first kinase structure to
be solved. The solution was achieved by a
combination of computational methods, in-
cluding refinement on the SDSC CRAY Y-
MP using the program XPLOR, developed
by Axel Bruenger of Yale University. Most
important to the solution was the ability of
the scientists to study .stereo visualizations of
the structure on a large screen at SDSC, so
scientists from every discipline within the
group could contribute their insight to a col-
lective determination of the structure. Kinases
play important messenger roles in cell me-
tabolism, and hundreds of such compounds
have been identified and sequenced. Becau.se
sequences are homologous in long stretches,
the solution for cAPK is proving extremely
valuable as a template for modeling and de-
riving the structure of other kinases. Taylor
and her group have collaborated with several
other groups since in modeling proposed
solutions for other kinases, including those
known to have carcinogenic properties or to
be involved in other disease processes. In all
of these studies, computation and visualiza-
tion have played an important role. Solutions
for various kinases can lead to the design of
inhibitors to prevent the enzymes from acting
to produce diseases. The work won the Fore-

fronts of Large-Scale Computation Award
presented at Supercomputing '93.

Molecular Neuroscience-Serotonin

A number of cardiovascular and psychiatric
diseases are currently treated with drugs that
act on the neurotransmitter serotonin and its
receptors. The cellular receptor for serotonin
is a gatekeeper molecule that recognizes and
binds the serotonin and then transmits the
signal to the cell by binding to a special class
of transducers: the G-proteins. Using the
CTC's ES/9000, Dr. Harel Weinstein,
chairman of the Department of Physiology
and Biophysics at Mount Sinai Medical Cen-
ter, made a breakthrough in modeling the se-
rotonin receptor. His breakthrough came
from modeling the structural changes that
occur in the serotonin receptor when it binds
to a ligand and to the G-protein. causing it to
carry out its function. This research shows
how G-proteins can be switched on by
structural changes in specific regions of the
receptor molecule and is expected to lead to
the development of more effective drugs,
specific ligands aimed at the regions where
the structural change takes place. Weinstein
believes that his work may be applied more
broadly to other receptor molecules, includ-
ing all neurotransmitters, that communicate
with cells via G-proteins. If Weinstein can
demonstrate a common mechanism of re-
spon.se in these receptors, he will have a new
type of molecular approach to treating a vast
range of diseases. Weinstein has also u.sed
the Cray Research C-90 and Intel Paragon at
SDSC.

Molecular Neuroscience-Acetyl-
cholinesterase

A collaboration between Michael Gilson,
T.P. Straatsma, and Andrew McCammon
(Dept. of Chemistry. University of Houston),
Daniel Ripoll (Research Associate. CTC),
Carlos Faerman (Dept. of Molecular and Cell
Biology, Cornell Univ.). Paul Axelsen (Dept.
of Pharmacology, University of Pennsylva-
nia School of Medicine), and Israel Silman
and Joel Sussman (Weizmann Institute of
Science, Israel) has used molecular dynamics

26-

High Performance Computing Infrastructure and Accomplishments

algorithms to investigate the rapid activity of
the enzyme acetylchoHnesterase (AChE). The
enzyme breaks down the neurotransmitter
acetylcholine diffused across nerve cell syn-
aptic gaps. Its three-dimensional crystal
structure was solved by Joel Sussman and
colleagues at the Weizmann Institute in Re-
hovot, Israel, several years ago. That struc-
ture showed the active site to be a long, nar-
row channel — too narrow to deal rapidly with
the job of dissociating acetylcholine into
choline and an acetate ion. Yet it is known
that AChE acts very rapidly, no doubt be-
cau.se of aeons of evolutionary pressure to
optimize the response of the nervous system
in all organisms. Molecular dynamics calcu-
lations performed on the MetaCenter's Cray
Research C-90 (SDSC and PSC), TMC CM-
5 (NCSA), and Kendall Square KSR-1
(CTC) showed that there was also a "back
door" to the active site, that might open to
facilitate the exit of the acetate ion from the
site. A study of the electromagnetic fields of
acetylcholinestera.se with the back door
closed and then open supplied confirming
evidence. Since inhibitors of AChE are im-
portant medications for myasthenia gravis,
glaucoma, and Alzheimers's disease, this
new insight may lead to more effective
pharmaceutical agents to fight these diseases.
This work was the cover story of the March
4, 1994 issue of Science magazine.

Kinking DNA

John M. Rosenberg (University of Pitts-
burgh) used the PSC Cray Research C-90
vector supercomputer to determine how a
protein identifies and interacts with specific
sites of DNA — a fundamental biological
process called "protein-DNA recognition,"
which is related to many disease processes
and is also a vital tool in the biotechnology
industry. Rosenberg's molecular dynamics
simulations have refined the structure of an
important protein, Eco Rl endonuclease,
u.sed in DNA cloning, and they have resulted
in a clear understanding of a "kink" in the
DNA backbone that results when Eco Rl
endonuclease binds with DNA. Rosenberg

won the 1991 Forefronts of Large-Scale
Computation award for this research, and his
work was cited in the 1993 Compute rwnrld
Smithsonian award for .science given to the
PSC.

Antibody-Antigen Docking

A collaboration among computer scientists
Michael Hoist and Faisal Saied (Dept. of
Computer Science, UIUC) and two biolo-
gists Richard Kozack and Shankar Subra-
maniam (Dept. of Physiology and Biophys-
ics. UIUC and Beckman Institute/NCSA)
has been able to solve for the first time the
complete nonlinear Poisson-Boltzmann
equation, which is the fundamental equation
of macromolecular electrostatics. A method
based on multigrid-inexact Newton algo-
rithms has been developed and large memory
applications run in parallel on the NCSA
Convex C3 show that this has profound con-
sequences for protein structure, enzyme
mechanisms and protein design. Coupling
this new approach with a Brownian dynam-
ics method, the largest simulation ever of an
encounter between two proteins, an antibody
and an antigen, has been carried out using the
NCSA CM-5 and SGI Challenge. This
simulation for the first time is able to give
rate constants for association of proteins that
is comparable to experimental measure-
ments. The results of the electrostatics work
was the cover story of the March 1994 issue
of Proteins: Structure, Function, and Ge-
netics.

Tuning Biomolecules to Fight Asthma

Over the last 20 years, the number of asthma
cases has almost tripled in the U.S. David
Herron, senior research scientist at Eli Lilly
and Company, is searching for new drugs
that will inhibit the action of leukotrienes, in-
flammatory agents released by several types
of cells in the lungs, which cause the lungs to
stiffen and become imtated. Several gi-
gabytes of data from molecular dynamics of
three key leukotrienes, run on NCSA's and
Lilly's Cray-2 supercomputer were analyzed
in a lengthy scientific visualization created
with NCSA staff. Using the animations as a

27-

High Performance Computing Infrastructure and Accomplishments

guide. Lilly asthma researchers synthesized
highly active antagonists against the leuko-
trienes. Some have been tested in asthma
sufferers and found to be effective medically.
For this work, Herron was a co-recipient of
the first NCSA Industrial Grand Challenge
Award.

Virtual Spider and Artificial Silk

Biophysicist Lynn Jelinkski, director of
Cornell University's Biotechnology Center
for Advanced Technology (CAT) is com-
bining medical imagmg techniques with the
state-of-the-art computer visualization re-
sources of the CTC to study the molecular
structure of the strongest silk of the golden
orb weaver spider and its transformation
from a viscous fluid into the extremely
strong cry.stalline fiber which has the poten-
tial to replace manmade fibers, such as nylon,
manufactured from petrochemicals. Jelinski
has devised a way to create a 3-D "computer
spider" by compiling stacks of the 2-D MRI
images using the IBM POWER Visualiza-
tion System, one of the high-performance
computing resources of the CTC. Each im-
age contains over 100,000 pixels. Hundreds
of images are combined to construct the 3-D
simulated .spider. Once in hand, this virtual
spider can be dissected by computer to de-
scribe the anatomy of the glandular system
and to provide the physical processing in-
formation Jelinski seeks. This understand-
ing, coupled with molecular-level studies of
the amirio acids that make up the web silk
polymer, may aid in genetically engineering
plants to produce fibers as strong as those
produced by the spider. Jelinski's work
blazes a path toward the development of a
new class textiles with superior strength at
the same time that it promises fundamental
insight into the mystery of the spider's web.

Heart Modeling

Charles S. Peskin and David M. McQueen
(Courant Institute, New York University)
have developed over the last decade a fully
functionmg three-dimensional model of the
heart, its valves and nearby major vessels.
This computational model will make it pos-

sible to study questions about normal and
diseased heart function that are difficult or
impossible to address through animal and
clinical studies. The complexity of the heart
model is so great that a single heartbeat re-
quires a 150 hour run on the PSC Cray Re-
search C90 and could not have been run
without the very large memory of the C90.
This research won the 1994 Computerworld
Smithsonian award for Breakthrough Com-
putational Science. Peskin was awarded a
Mac Arthur Prize Fellowship in 1983.

Engineering

Man-made devices have become so complex
that researchers in both academia and indus-
try have turned to supercomputers in order to
be able to analyze and modify accurate mod-
els in ways which complement the traditional
experimental methods. Such easily accessible
high performance computers enable aca-
demic engineers to study the brittleness of
new types of steel, to improve bone trans-
plants, or to reduce drag of flows over sur-
faces using riblets. Industrial partners of the
individual supercomputer centers within the
MetaCenter are using computational facilities
more advanced than they have access to in-
temally to improve indu.strial proces.ses such
as in metal forming. Better consumer prod-
ucts such as leakproof diapers, or more effi-
cient airplanes are being designed. Even
State agencies are able to use the MetaCenter
facilities to improve traffic safety or find bet-
ter ways to use recycled materials. Some 70
corporations have taken advantage of the
MetaCenter industrial programs to improve
their competitiveness.

Ultra-high-strength Steels

28-

High Performance Computing Infrastructure and Accomplishments

molecular mechanisms that give rise to im-
purity-mduced embrittlement in steel, work
which IS expected to lead to steel that will not
shatter in frigid conditions. This work was
reported in the July 15, 1994 Science. Free-
man and his group have been users of NSF
supercomputers since the founding of the
program on a wide range of problems in
materials sciences. In recognition of his pi-
oneering work in computational materials
research. Freeman received the first Materials
Research Society Medal and the first Award
in Magnetism from the lUPAP.

Continuous Casting of Steel

Achilles Vassilicos (U.S. Steel Technical
Center) models the flow of molten steel in a
continuous-casting "tundish" on the PSC
Cray Research C90, resulting in improved
process control over the quality of steel. By
more accurately predicting the precise metal-
lurgical composition of the continuous-
casting output "slabs," U.S. Steel reduces
waste steel and the amount of inventory it
must keep on-hand, resulting in substantial
cost savings.

Beverage can design

Three-dimensional stress modeling of alumi-
num beverage cans on the PSC C90 by Rob-
ert E. Dick and Andrew B. Trageser
(ALCOA Laboratories) has greatly reduced
the expense of developing a new can design
that will meet customer specifications for
strength and appearance. By relying less on
costly, time-consuming prototype testing,
ALCOA engineers estimate a co.st savings of
SI 00,000 or more per can design. This re-
search has been described in articles in D/,?-
crnrr (March 1991), Business Week (Oct. 8,
1990) and in Science (June 23. 1989).

Designing a Leakproof Diaper

Designing effective and comfortable dispos-
able infant diapers requires greater un-
derstanding of the function of the diaper
components, such as the cellulosic fluff and
the superabsorbent polymer particles-and the
effect of variations of parameters related to
these components. Dow Chemical Company.

has done extensive experimental testing,
evaluation, and computer modeling that has
contributed to a faster developmental process
and shortening the time for new product in-
troduction. The innovative Dow design was
evaluated using a computer model run on
NCSA's CRAY-2 system. Three separate
time-dependent processes are modeled. The
first, a fast spreading process, involves the
insult on the pad by a quantity of liquid
(urine) which is transported through the pad
by wicking. During this process the fluff pad
collapses as it becomes wet. The second,
called the imbibition process, models the
swelling of the superabsorbent polymer par-
ticles and their uptake of liquid from the cel-
lulose fluff. During this process, the fluff ex-
pands again. The slowest-and final-process
tracks the redistribution of liquid in the fluff
pad as the saturation of the fluff adjacent to
the superabsorbents changes. The overall
model was compared to a magnetic reso-
nance imaging experiment, which provides a
three-dimensional image of the water distri-
bution in a diaper, and was shown to give
comparable results to the final steady-state
values. Optimization of these processes is
leading to an improved, quality diaper.

Bone Transplant Bioengineering

Dean Taylor and Donald Bartel (Dept. of
Mechanical and Aerospace Engineering,
Cornell University) have been able to investi-
gate bone-implant systems across a wide
range of design parameters by using high
performance parallel computing (including
the IBM SPI) and visualization resources at
CTC. Their long-term research has produced
models of the stresses placed on normal
bones and the artificial components of a hip
joint — these models have led to custom-
designed prostheses and reduced prosthesis
replacement surgery.

Improving Performance with Riblets

George Em Karniadakis' and his group at
Brown University are using the SDSC Intel

Uncidentally, Prof. Karniadakis is the current
chair of the joint NCSA /PSC National Peer

-29-

High Performance Computing Infrastructure and Accomplishments

Paragon to explain drag reduction in turbulent
flow by means of "riblets," parallel grooves
on the surface of an object moving through
the flow. Such grooves are used on aircraft,
in pipelines, and on racing cars and sleds, to
improve performance. The group also mod-
els flows in micro-electromechanical systems
used in surgery and other complex applica-
tions, where the molecules of the fluid are not
much smaller than the channels in which they
flow. Both projects are resulting in new ways
to optimize the performance of broad classes
of machinery.

Designing Better Aircraft

Dino Roman, John Vassberg, and Tom
Gruschus of McDonnell Douglas are using
SDSC's Cray C-90 for a computational fluid
dynamics simulation of an aircraft in flight
and to visualize the results using FAST
(Flow Analysis Software Toolkit) on a Sili-
con Graphics IRIS workstation. Tracer parti-
cles are released into the flow field in front of
the aircraft and allowed to follow the stream-
lines around the vehicle. A cutting plane
through the 3D volume of data is placed to
intersect the aircraft fuselage and wings. The
aerospace industry relies on computational
fluid dynamics-the simulation of air or fluid
flow-to design, develop, and test new aero-
nautical configurations. This process enables
companies to test new models quickly to .se-
lect candidates for wind-tunnel testing. Such
methods used in aircraft design and manufac-
ture can give American companies a techno-
logical edge in the global market.

Crash Testing Street Signs

California State Department of Trans-
portation (Caltrans) engineer Payam Row-
hani and SDSC engineer Chuck Charman
generated a computerized model of a crash-
test vehicle, called a "bogie," on the SDSC
Cray Research Y-MP for simulations of test
crashes with sign and lighting supports. They
fine-tuned the bogie front-end design with the

Review Board, another example of the tight
luiks between members of the MetaCenter and
the scientific community.

computer model to minimize the number of
validation tests necessary at the Federal Out-
door Impact Laboratory, operated by the
Federal Highway Administration. In a sec-
ond application, Charman is working with
Caltrans engineers William Nokes and Dario
Perdomo, who are designing pavement using
structural modeling techniques. They are re-
searching the use of new and recycled mate-
rials and these materials' response to differ-
ent axle and tire configurations. They are us-
ing the supercomputer and the visualization
facilities to explore new truck suspension
systems, and new tires and heavier loadings,
innovative pavement structures with recycled
materials and rubber and polymer-modified
binders. The results of this research are ex-
pected to lead to significant cost savings in
the design, construction, maintenance, and
rehabilitation of pavement structures.

Earth Sciences and the En-
vironment

From understanding the motions of the
Earth's convective mantle to daily compu-
tation of air pollution levels in southern Cali-
fornia, the resources of the NSF MetaCenter
are being used to compute and visualize the
complexity of the natural world around us.
The US Army is working with academics to
determine how they can practice tank maneu-
vers without endangering the breeding habits
of the sage grou.se. Pollution, whether under-
ground or in the air, is a difficult coupling of
chemical reactions and flow dynamics which
must be understood in detail if corrective
measures are to be efficacious. High per-
formance computers also act as time ma-
chines, allowing for faster-than-realtime
computation of severe storms. Finally, to
improve global weather or climate forecasts,
supercomputers allow researchers to zero in
on the critical coupling physics of such proc-
esses as mixing at the air/ocean interface.

Detoxiflcation of Ground Water

Christine Shoemaker (Dept. of Civil and En-
vironmental Engineering, Cornell University)
has been a pioneer user of the scalable IBM
SP machines at the CTC for the development

30-

High Performance Computing Infrastructure and Accomplishments

of numerically efficient supercomputer algo-
rithms for optimal control of dynamical sys-
tems and the application of these techniques
to detoxification of contaminated groundwa-
ter. Her efforts are leading to methods of
determining the most cost-effective way to
clean up the groundwater by computing time-
varying rates of pumping. Shoemaker's
group has also developed an animation, using
the visualization resources (both hardware
and personnel) of the CTC, that represents
the effects of different policies and natural
chemical and biological processes on
groundwater cleanup. Such animations are
crucial for conveying the results of basic re-
search to the mixed audience involved in set-
ting environmental policy.

Sage Grouse-Endangered Species and
the US Army

Working with the U. S. Army, Bruce Han-
non (Dept. of Geography, UIUC) and Jim
Westervelt, of the U. S. Army Construction
Engineering Research Laboratory, have de-
veloped an ecological model for the sage
grouse, an endangered species, population on
an army training base in Washington State.
Using the Macintosh software STELLA, the
CM-5 and the GRASS geographical infor-
mation systems, these researchers were able
to optimize the scheduling of training exer-
cises to maximize grouse reproduction and
longevity. For each geographic cell, a very
large STELLA model was constructed, rep-
resenting the grouse at various life stages,
different kinds of plants and predators, soil
type and moisture, weather-all the many
physical variables-and also introduce the nec-
essary human activities like tank and troop
maneuvers on the army range. Each cell, of
which there are over a hundred thousand in
the GIS covering the army base, could con-
tain 1 00 to 200 variables. This work demon-
strated the efficacy of coupling GIS datasets
to ecological models and running them in a
client-server fashion between a Macintosh
and the CM-5.

Storm modehng/forecasting

Robert Wilhelmson (Dept. of Atmospheric
Sciences. UIUC and NCSA) and his col-
leagues have been able to simulate the devel-
opment of tornadoes embedded within larger
storms called supercells (producing the larg-
est tornadoes) and along low level con-
vergence boundaries (e.g. along a thunder-
storm cold air boundary) using both tradi-
tional vector supercomputers (NCSA and
PSC Cray Research Y-MP and C90 and
NCSA TMC CM-2 and CM-5). Study of
these results is leading to a better under-
standing of when tornadoes will develop and
to more accurate tornado warnings. The visu-
alization of the intemal dynamics of a .severe
thunderstorm, created by the NCSA visuali-
zation team in 1989, is perhaps the most
widely viewed visualization of a supercom-
puter simulation ever made. It had a major
worldwide impact on the adoption of scien-
tific visualization as a working tool of com-
putational science.

Kelvin Droegemeier, a former student of
Wilhelmson's, and his colleagues associated
with the Center for Analysis and Prediction
of Storms (CAPS), an NSF S & T Center,
have used the NCSA and PSC Cray Re-
search supercomputers to develop the Ad-
vanced Regional Prediction System, a com-
putational model for forecasting severe
storms. As of Spring 1994, this model has
been used, with data augmented by the sin-
gle-Doppler radar network now being de-
ployed by NOAA, in daily weather reporting
on an experimental basis. Because of their
use of parallel supercomputers they have
shown that regional storm forecasts based on
very high resolution models are possible with
the advent of teraflop computing capabilities
in the next few years. The long-term objec-
tive is to improve the prediction of hazardous
weather on scales ranging from a few kilo-
meters (an individual storm) and tens of
minutes to hundreds of kilometers (a squall
line or other mesoscale system) and several
hours.

High Performance Computing Infrastructure and Accomplishments

Los Angeles Smog

Gregory J. McRae (Massachusetts Institute
of Technology) and Armistead Russell
(Carnegie Mellon University) have developed
the most comprehensive model of smog
formation available. Their modeling of smog
in Los Angeles on the PSC C90 showed that,
contrary to EPA policy at the time, it is nec-
essary to control nitrogen oxide emissions as
well as hydrocarbons to control smog. This
work formed the scientific underpinning for
the Air Quality Management Plan adopted in
1988-89 for the Los Angeles air basin, the
most stringent such plan in the United States.
Their modeling also showed that alternative
vehicle fuels, methanol in particular, repre-
sent a worthwhile strategy for improving ur-
ban air quality, which influenced inclusion of
this policy in the 1990 revisions to the Fed-
eral Clean Air Act. This work is being ex-
tended using the combination of the PSC
Cray Research C90 and T3D as an NSF
Grand Challenge. The first Forefronts of
Large-Scale Computation award, given in
1989, recognized McRae for this work.

These pioneering computations are leading to
practical tools for states to predict air pollu-
tion levels. The Modeling and Meteorology
Branch of the California Air Resources
Board joined the SDSC Industrial Partners
program in 1991. They are running, on the
SDSC Cray Research C90, the Urban Air-
shed model which estimates hourly pollutant
concentrations. They results are u.sed to es-
timate maximum pollutant concentrations or
population exposure statistics for different
emissions controls.

Upper Ocean Mixing

Sidney Leibovich (Dept. of Mechanical and
Aerospace Engineering, Cornell University)
has developed a mathematical model on the
CTC IBM ES/9000 and SPl that uses a .se-
ries of equations to simulate what happens
when wind blows, waves form, and Lang-
muir circulations begin to mix the upper
ocean. This mixing has a marked influence
on the density of the water; the ocean's den-
sity structure alters its current patterns, and

the combined phenomena influence the ex-
change of heat between the ocean and the at-
mosphere. All of this has environmental and
ecological consequences, and finding ways to
predict this chain of events is of interest to
ecologists. oil companies, meteorologists,
climatologists, undersea communications
experts, and government policy makers.

Simulating Climate using Distributed
Supercomputers

C. Roberto Mecho.so, University of Cal-
ifornia, Los Angeles; Larry Bergman, Jet
Propulsion Laboratory; Carl Scarbnick, Gary
Hanyzewski, and Bilal Chinoy, SDSC; Paul
Messina, California Institute of Technology;
and Robert Malone and Rick Smith, Los
Alamos National Laboratory are developing a
coupled general circulation model distributed
over a heterogeneous network. The atmos-
pheric component, which runs on JPL's
CRAY, is a finite-difference model with
state-of-the-art parameterizations of convec-
tion, planetary boundary layer processes, and
radiation. It provides wind stress and heat
flux information to the finite-difference oce-
anic component, which runs on Caltech's
Delta or SDSC's Paragon. The oceanic com-
ponent then returns the sea surface tempera-
ture to the atmospheric component. Results
of the coupled system show a very realistic
seasonal cycle. This is a significant step in the
development of a coupled system distributed
across heterogeneous computer environ-
ments, a mam goal of the CASA gigabit test-
bed project.

Planetary Sciences, astronomy,
AND Cosmology

The sciences of the world beyond the Earth
have always been an interaction of ob-
servation of unexpected events and theory
creating a model built on the laws of physics
and chemistry which explain the observa-
tions. Therefore, it is proper to see high per-
formance computing and communications
making major impacts on both observation
and theory. Indeed, as we saw in the recent
impact of Comet Shoemaker-Levy 9 with
Jupiter, the observatories on earth and in

32-

High Performance Computing Infrastructure and Accomplishments

space have become intimately linked with
each other and with the theoretical simula-
tions. Supercomputers are becoming inte-
grated into observational facilities, like the
Grand Challenge BIMA millimeter observa-
tory, and with observational programs like
the ones which have led to discovery of new
millisecond pulsars or the first extrasolar
system planets, themselves orbiting a pulsar.

The ability of numerical methods to solve
even the most complex of fundamental
physical laws, such as Einstein's equations
of General Relativity, is leading to a very
rapid understanding of the dynamics of
strong field events, such as the collision of
black holes. In perhaps the grandest .scale
challenge possible, the universe itself is a
subject of investigation by a several Grand
Challenge teams using resources of the
MetaCenter to discover how the large scale
structures in the universe evolved from
nearly perfect homogeneity at the time of the
formation of the microwave background.

Comet Collision with Jupiter

Mordecai-Mark Mac Low (Dept. of As-
tronomy & Astrophysics, Univ. of Chicago
and Dept. of Astronomy, UIUC) and Kevin
Zahnle (Space Science Division, NASA
.Ames Research Center) simulated the impact
of Comet Shoemaker-Levy 9 on Jupiter us-
ing the astrophysical gas dynamics code
ZEUS-3D, developed at the NCSA, running
on the PSC Cray Research C90. Their simu-
lations, made months before the actual im-
pact, were widely used by planetary as-
tronomers to prepare for their observing
runs. Their results showed that the comet
would penetrate only shallowly into the
planet's atmosphere before exploding and
that the explosion would break out of the at-
mosphere. Their animations were available
using the global MosaicAVWW infrastruc-
ture and received worldwide coverage in both
the popular and scientific press.

Discovery of First Extrasolar System
Planet

A statistical analysis by Alexander Wol-
szczan (Dept. of Astronomy and Astro-
physics, Penn State Univ.) of data received
from the radio telescope at Arecibo Ob-
servatory in Puerto Rico led him to theorize
the existence of planets orbiting around a pul-
sar. Searching millions of bits of informa-
tion using the supercomputer facilities at the
CTC, he detected an unusual complexity in
the pattern of the pulses' arrival time. The
behavior suggested that the pulsar's motion
is affected by the presence of other orbiting
objects. Applying a perturbation model pro-
posed by Frederic Rasio at Princeton Univer-
sity, along with Philip Nicholson, Stuart
Shapiro, and Saul Teukolsky at Cornell Univ.
to a new set of observations, Wolszczan
found that it predicted the effect that two or-
biting planets would have upon each other as
they moved through space. The model fits,
and Wolszczan can be credited with identify-
ing the first planetary system beyond our
own.

The April 22 1994 issue of Science featured
this work, with front page coverage of news
also in the New York Times that week as well
as several other major metropolitan newspa-
pers, television, and popular magazines.
Wolszczan continues to explore this new
world using parallel capabilities of the IBM
SP machines at the CTC.

Building the BIMA Radio Telescope

This HPCC Grand Challenge project at
NCSA is implementing a prototype of the
next generation of astronomical tele.scope
systems - remotely located telescopes con-
nected by high-speed networks to very high
performance, scalable architecture computers
and on-line data archives, which are accessed
by astronomers over Gbit/sec networks. Data
taken by the BIMA telescope in northern
California are now transferred in real time to
NCSA (using a combination of high-speed
leased line and NSFnet), entered into a data-
base program, and archived on the NCSA
mass storage system. Astronomers locate

33-

High Performance Computing Infrastructure and Accomplishments

data in the archive and load data onto NCSA
or remote computers for processing using the
NCSA Mosaic software. Also, a digital li-
brary of fully processed radio images, which
will mclude an all-sky radio survey created
by the NRAO's VLA with a size of 1
terabyte, is being developed. One of the ma-
jor activities that BIMA and NCSA staff are
involved in is the creation of a new standard
software package AIPS-i-i-, written in C++
for extensibility and maintainability, to re-
place the older standard packages AIPS,
which was written in FORTRAN. The
AlPS-t-t- software is being developed by an
international collaboration including BIMA
member institutions, the National Radio As-
tronomy Ob.servatory. and scientists from
Australia. Canada. England, the Netherlands
and India. With AIPS-)-+, an astronomer can
learn the system once and then process im-
ages on nearly any computer using data from
almost any radio telescope. NCSA will im-
plement computationally intensive routines
on the SGI Power Challenge. Finally, discus-
sions are underway to have the BIMA tele-
scope array be part of the development of a
very large national millimeter array radio
telescope, in cooperation with the Caltech and
the National Radio Astronomy Observatory.

FuLsar Searching and Discovery

Joseph Taylor (Dept. of Astrophysical Sci-
ences, Princeton Univ. and 1993 Nobel prize
winner in Physics) and his colleagues used
the PSC C90 to aid in the search for pulsars
by analyzing radio-frequency data gathered at
the Arecibo Observatory. Since using su-
percomputing to sift through the enormous
amount of data obtained, this approach has
resulted in the discovery of 20 previously
unknown pulsars, including five millisecond
pulsars.

Accretion Disks Around Black Holes

Most galaxies .seem to harbor massive black
holes (between one million and ten billion
solar masses) at their centers. When gas ac-
cretes onto these holes enormous energies
can be released, turning the most luminous of
these objects into quasars. John Hawley

(Dept. of Astronomy. Univ. of Virginia) has
been using supercomputers for over a decade
to investigate the accretion disks which form
around black holes. Using the Cray Research
Y-MP and C90 at NCSA and PSC. Hawley
developed a new magnetohydrodynamic
(MHD) code which enabled him and Steven
Balbus to explore the nature of a powerful
MHD instability in accretion disks. They
discovered that this almost universal instabil-
ity is the underlying physical mechanism
which gives rise to the effective turbulent vis-
cosity in the disk, which previously theorists
had only modeled parametrically. For this
discovery. Hawley received the 1993 Helen
B. Warner Prize from the American As-
tronomical Society.

Black Hole Collision Dynamics

The two-body problem in General Relativity
is represented by the motion of two black
holes. In contra.st to the two-body problem
in Newtonian gravity which was solved in
closed form over 300 years ago. the orbiting
motion of two black holes is an outstanding
research frontier, recently being funded by
the NSF as a Grand Challenge. Two re-
search teams in this Grand Challenge. Ed
Seidel, NCSA Director Larry Smarr. and
coworkers (Dept. of Physics and NCSA,
UIUC) and Stu Shapiro and Saul Teukolsky
(Dept. of Astronomy and Physics, Cornell
University) have used a variety of supercom-
puters at NCSA. PSC. and CTC to compute,
for the first time, the evolution of the event
horizon representing the head-on collision of
two black holes. The spacetime diagram of
the collision shows a picture like the pre-
dicted "pair of pants", analyzed qualitatively
by Stephen Hawking and others in the early
1970s, but the supercomputer computation
gives for the first time quantitative details
about the area increase and the caustics
formed as the null geodesies, which generate
the horizon, intersect. Seidel and coworkers
show that in general such violent deforma-
tions of black holes cause the horizon to os-
cillate with the quasinormal modes previ-
ously studied only in linear perturbatioiT the-

■34

High Performance Computing Infrastructure and Accomplishments

ory. The Grand Challenge team is now mov-
ing on to the full spiraiing-in situation, which
is exjjected to be one of the first events ob-
served by the LIGO gravitational wave ob-
servatory late in this decade.

Largest cosmological simulation

This simulation, carried out by Michael
Norman and Greg Bryan (NCSA and Dept.
of Astronomy), on the TMC CM-5 is the
first which is sufficiently accurate to allow
direct comparison with x-ray telescopic ob-
servations, thereby allowing a critical test of
this popular cosmological model. A block of
the universe half a billion light years on a side
was evolved from the era of the formation of
the microwave background (with tiny inho-
mogeneties as quantified by the COBE satel-

lite) until the present. A single evolution re-
quired 30 hours on the 512-node CM-5 and
over 14GB of memory to hold the model.
The results were announced at the May 1994
meeting of the American Astronomical Soci-
ety and covered in the popular press by CBS-
TV and the Washington Post. The computa-
tion is one of a number now being made by
members of the Grand Challenge Cosmol-
ogy Consortium using resources of the
MetaCenter. Jerry ©.striker (NAS), Chair of
the Department of Astrophysical Sciences at
Princeton University is The largest simula-
tion of the formation of x-ray clusters in a
universe dominated by a mixture of cold and
hot dark matter was carried out using
NCSA's 512 node CM-5. PI of the GC-"*.

Evolution of the MetaCenter Concept

As the decade of the 1990s opened, it became
clear that we were witnessing fundamental
changes in the way researchers use high-
performance computing and communications
systems, driven by technological advances in
computers and networking. Anticipating
these changes, the four NSF supercomputing
centers formed a collaboration based on the
concept of a national MetaCenter for compu-
tational science and engineering: a collective
of intellectual and physical resources unlim-
ited by geographical or technological con-
.straint.

The primary objective was to realize ways in
which researchers could be best served and
their problems directed to that architecture or
combination of architectures best suited to
their solution, on a nationwide basis. Thus
the MetaCenter. begun in 1992, was con-
ceived as a "garden of architectures" that
could demonstrate ways of providing na-
tionwide resources to support scientific and
technological advances and supply a frame-
work within which the NSF centers program
could move into the next era of high-
performance computing. In 1994. NCAR's
Scientific Computing Division joined the
MetaCenter. In addition, the NSF established

a program of MetaCenter Regional Affiliates
(MRA) under which other institutions could
pursue projects of interest in collaboration
with MetaCenter institutions; MetaCenter
sites were involved with several MRA pro-
posals.

MetaCenter projects involve the use of the
resources and expertise available at more than
one MetaCenter site. Often these projects are
grand challenges or national challenges-
many have been highlighted el.sewhere in this
report.

In addition to direct support of researchers,
the MetaCenter sites have joined to work on
specific projects of benefit to the .scientific
community. Projects begun by the Meta-
Center during the first two years include;

• National File System; Separate cells
have been established at each site based on
the Andrew File System (AFS); a national
cell that will serve as the basis for a pro-
duction system is being implemented
With such a system in place, users will be
able to move transparently among the
MetaCenter computational systems.

-35-

High Performance Computing Infrastructure and Accomplishments

National Archival Storage System;
Work has begun on a proposed archival
storage system based on manufacturer-
supported and MetaCenter-enhanced prod-
ucts, including versions of UniTree and
the Data Migration Facility from Cray Re-
search. With such a system in place, files
on any center machine can be migrated
transparently among other center ma-
chines, the national file system, and the na-
tional archival .storage system.

Video Collaborator/: Using video
teleconferencing equipment, the re-
searchers and staff of MetaCenter in-
stitutions meet regularly, often with par-
ticipation from the NSF. The equipment
has been used to broadcast joint training
workshops like the MetaCenter Computa-
tional Science Institutes in Parallel Com-
puting conducted at SDSC and NCSA.
Plans" are to install additional hardware to
strengthen this system.

Visualization Collaboration: Meta-
Center and NSF Science and Technology
Centers researchers participated in a
DASC-funded MetaCenter Visualization
Workshop held at SDSC in 1993, and ex-
periments have begun in passing visual-
izations between the centers in real time to

determine bottlenecks and potential prob-
lems.

Online Information: An online in-
formation system using the gopher proto-
cols has been implemented; it is now be-
ing converted to be accessible via World-
Wide Web browsers. Plans are to broaden
the information available and incoiporate
new technologies. A joint Science High-
lights Repository will soon include
WWW links to MetaCenter-supported re-
search projects worldwide.

• Networking: Plans were developed to
upgrade the high-speed, wide-area net-
work connecting the centers into a very
high speed backbone.

New projects for the MetaCenter include ex-
ploring virtual reality for use in the scientific
discovery process, u.se of the basis developed
in the Online Information project to develop a
national training program to promote under-
standing and expertise in the use of parallel
computing, and a joint evaluation of the
scalable parallel computers acquired by the
NSF centers. A project has also been initiated
to establish a security research and response
team to monitor and coordinate responses to
incidents of .security violations.

Recognition Accorded NSF supercomputer Users and P r oj ects

1994 [in part for this work performed at NSF
centers]. David's PhD advisor and the third
Feenberg awardee is Mai Kalos, Director of
the Cornell Center.

The previous sections on Accomplishments
of the NSF Supercomputer Program cite
several major ways in which the research of
the centers has been recognized. Specific no-
table citations and users are summarized
here, with the complete description available
in the appropriate sections. In addition, scien-
tific visualizations produced at the centers
have provided popular recognition as covers
of scientific magazines and technical pro-
grams, as evidenced by numerous covers of
Science, and NSF's own publication Mosaic.

Pha.se Transitions of Solid Hydrogen ...
David Ceperley was awarded the fifth Eu-
gene Feenberg Memorial Silver Medal in

Protein Kina.se solution ... (Susan Taylor et
al) won the Forefronts of Large-Scale Com-
putation Award presented at Supercomputing
'93 for this work...

Kinking DNA ... John Rosenberg won the
1991 Forefronts of Large-Scale Computation
award for this research, and his work was
cited in the 1993 Computerworld Smith-
sonian award for science which was given to
the Pittsburgh Supercomputer Center (PSC).

-36-

High Performance Computing Infrastructure and Accomplishments

Heart Modeling ... This research by Charles
Peskin won the 1994 Computerworld Smith-
sonian award tor Breakthrough Computa-
tional Science. Peskin was previously
awarded a Mac Arthur Prize Fellowship in
1983, for earlier studies in heart modelmg.

Ultra-high-strength Steels Gregory B. Olson
and Arthur J. Freeman (Northwestern Uni-
versity) use computer modeling to design
ultra-high strength steel for weight-critical
applications ... This work was reported in the
July 15. 1994 Science. Freeman and his
group have been users of NSF su-
percomputers since the foundmg of the pro-
gram on a wide range of problems in materi-
als sciences. Arthur Freeman received the
first Materials Research Society Medal and
the first Award in Magnetism from the Inter-
national Union of Pure and Applied Physics.

Los Angeles Smog ... The first Forefronts of
Large-Scale Computation award, given in
1989, recognized Gregory McRae for this
work.

Theory of High Temperature Super-
conductors ...The Nobel Prize in Physics in
1 987 was for the discovery of a new class of
high temperature superconductors. Thou-
sands of research papers have been written
about the.se unique materials... David Pines
(NAS and first Feenberg 'Medalist) and
Philippe Monthoux (Department of Physics,
UIUC) used the NCSA Cray Research Y-
MP to carry out a strong coupling
(Eliashberg) calculation of the normal state
properties and Tc for the model experiment-
based magnetic interaction between quasipar-
ticles...

Crystallography... Herbert Hauptman
(Medical Foundation of Buffalo. Inc.) won a
Nobel Prize in 1985 for development of the
"direct method" of protein structure de-
termination from X-ray crystallographic data.
In a collaboration with Russ Miller (State
University of New York at Buffalo). Haupt-
man and Miller have developed a numerical
approach that extends the "direct method" of
determining molecular structure from X-ray

crystallographic data to larger molecules, be-
yond its present limit of about 100 atoms...

Pulsar Searching and Discovery ... Joseph
Taylor (Dept. of Astrophysical Sciences.
Princeton and a 1993 Nobel prize winner in
Physics) and his colleagues used the PSC
C90 to aid in the search for pulsars by ana-
lyzing radio-frequency data gathered at the
Arecibo Observatory. Using supercomputing
to sift through the enormous amount of data
obtained, this approach has resulted in the
discovery of 20 previously unknown pulsars,
including five millisecond pulsars.

Accretion Disks Around Black Holes ... John
Hawley (Dept. of Astronomy. Univ. of Vir-
ginia) has been using supercomputers for
over a decade to investigate the accretion
disks which form around black holes. Using
the Cray Research Y-MP and C90 at NCSA
and PSC, Hawley developed a new magne-
tohydrodynamic (MHD)code which enabled
him and Steven Balbus to explore the nature
of a powerful MHD instability in accretion
disks. ... For this discovery, Hawley re-
ceived the 1993 Helen B. Warner Prize from
the American Astronomical Society.

Alpha Shapes, Biomolecules, and Cosmol-
ogy Alpha shapes, a form of geometric mod-
eling developed by the 1993 Waterman
Award winner Herbert Edelsbruner (Dept. of
Computer Science, UIUC) and NCSA staff
member Ping Fu. focuses on the formal def-
inition, construction, and measurement of
shapes for any given point set in space, [They
are...] useful in studying water molecules re-
siding inside a protein. NCSA u.sers have
discovered other related applications of alpha
shapes by applying them to such diverse
fields as adaptive grid generation, medical
image analysis, visualizing the structure of
earthquake data, and the large-scale structure
of the universe.

The Rise of the MosaicAVWW Information
Infrastruture NCSA developed the Mosaic
user interface software which provides point-
and-click access to the diverse information
storage protocols of the Internet, such as
World Wide Web (WWW), Gopher, FTP.

37-

High Performance Computing Infrastructure and Accomplishments

and WAIS.... In 1994. NCSA was given In-
joworld's Publisher's Industry Achievement

Award.

Acronyms

AFS

Andrew File System

(developed at Carnegie-Melon

University and named for its

CFS

benefactors, Andrew Carnegie

CMRR

and Andrew Melon)

ARPA

Advanced Research Project
Agency of the Department of

CNRI

Defense (previously called

CPU

DARPA)

CTC

ASC

Division of Advanced Scien-

CTSS

tific Computing, Computer

DASC

and Information Sciences and

DEC

Engineering Directorate

(CISE). NSF

DICE

ATM

Asynchronous Transmission

Mode - high bandwidth

DJMS

transmission technology for

DMF

WAN use

BIMA

Berkeley, Illinois, MAryland
telescope

EOSDIS

BLANCA

NSF Gigabit testbed - see

CASA

EPA

A programming language de-

veloped at Bell Labs

FAST

(probably the 3rd in a series)

and used as an easily portable

FDDI

systems development lan-

guage.

C++

An Object Oriented extension
ofC

FORTRAN

CAD

Computer Aided Design

CAPS

Center for Analysis and Pre-
diction of Storms

CASA

NSF Gigabit testbed - see

FTP

BLANCA

CAT

Cornell University's Biotech-

Giga
GIS

nology Center for Advanced

Technology

CAVE

Cave Automated Virtual En-
vironment

Gopher

CCMS

Computational Center for

Macromolecular Structure —

HDF

an SDSC, UCSD, and

Scripps Research Institute
collaboration
Compact Disk
Common File System
Center for Magnetic Record-
ing Research

Corporation for National Re-
search Interests
Central Processor Unit
Cornell Theory Center
Cray Time Sharing System
see ASC

Digital Equipment Corpo-
ration

Distributed Collaboration En-
vironment)

Dynamic Job Mix Scheduler
Data Migration Facility (Cray)
Data eXplorer

NASA Earth Orbitting Satel-
lite Distributed Information
System

Environmental Protection
Agency

Flow Analysis Software
Toolkit

Fiber Data Distribution Inter-
face (a 100 Mps network me-
dium)

FORmula TRANslator - Ear-
liest high level programming
language — still in common
use on high performance
computers

File Transfer Protocol
Grand Challenge
One Billion- 10^
Geographic Information Sys-
tem

Internet Browser developed at
the University of Minnesota
(Golden Gophers)
NCSA Hierarchical Data
Format

38-

High Performance Computing Infrastructure and Accomplishments

HiPPI

High Speed Peripheral Paral-
lel Interface

NCAR

Hewlett Packard

NCSA

HPC++

An object oriented varient of

the C language for High Per-

NIH

formance Computers

Nil

HPCC

High Performance Com-

puting and Communications

NOAA

HPF

High Performance FOR-

TRAN

NRAO

HPSS

High Performance Storage

System

NREN

HTTP

Hyper-Text Transfer Protocol

for the Internet

NSF

IBM

International Business Ma-

NSL

chines

PICT

IFS

AFS port by Univ. of Michi-

gan

PSC

InterNIC

Internet Network Information

Center

PVM

lUPAP

International Union of Pure

QCD

and Applied Physics

RAID

IVEM

Intermediate Voltage Electron

Microscope

REL

LANL

Los Alamos National Labo-

latory

REU

LAN

Local Area Network

LIGO

Laser Interferometry Gravita-
tional Observatory

RISC

LLNL

Lawrence Livermore National

ROM

Laboratory

SDF

MacCHESS

Cornell High Energy Syn-
chrotron Source for Macro-

SDMIR

molecular Modeling

SDSC

MHD

Magneto HydroDynamics

MILC

MIMD Lattice Calculations

SGI

Collaboration

SIGGRAPH

MIMD

Multiple Instruction, Multiple
Data

Mosaic

Internet Hypertext Browser

— WWW client developed at

SIMD

NCSA

MPP

Massively Parallel Processors

STM

MRA

MetaCenter Regional Affiliate

MRI

Magnetic Resonance In-
terferometry

TCP/IP

NAS

National Academy of Sci-
ences

Telnet
Tera

National Center for At-
mospheric Research
National Center for Su-
percomputer Applications
National Institutes of Health
National Information In-
frastructure

National Oceanic and Atmos-
pheric Administration
National Radio Astronomical
Observatory

National Research and Edu-
cation Network
National Science Foundation
National Storage Lab
A popular image format for
computers

Pittsburgh Supercomputer
Center

Parallel Virtual Machine
Quantum ChromoDynamics
Redundant Array of In-
expensive Disks
Renaissance Experimental
Laboratory

Research Experience for Un-
dergraduates (NSF program)
Reduced Instaiction Set
Computer
Read Only Memory
Standard Data Format
San Diego Microscopy and
Imaging Resource
San Diego Supercomputer
Center

Silicon Graphics, Inc.
Special Interest Group -
Graphics — an influential and
popular visualization organi-
zation and annual trade show
Single Instruction, Multiple
Data

Scanning Tunneling Mi-
croscope

Tranmission Control Pro-
tocolAnternet Protocol
Internet Computer logon
Services
One trillion- 10'2

39-

High Performance Computing Infrastructure and Accomplishments

TMC

UNICOS

UNIX

vBNS

VPLA

Thinking Machines. Corp.
Cray'.s Version of Unix
Popular portable computer
operation system developed at
Bell Labs - reportedly does
not stand for anything - al-
though some believe that it
was a pun on Multics, a time
sharing systems then in use at
MIT

very high speed Backbone
Network Service
Visual Programming Lan-
guage for Animation

Virtual Reality

WAIS

Wide Area Information Sys

tem

WAN

Wide Area Networks

WWW

World Wide Web - a set of

commications protocols for

Hypertext established at

CERN.

XMP & YMP - versions of Cray computer
systems — vector multi-
processors— used as refer-
ences for computer perform-
ance

40-

Mr. SCHIFF. Thank you, Dr. Hayes.

I would like to recognize Members for questioning. I would ap-
preciate it if Members would try to stay at least loosely to five min-
utes just because we have two more panels.

I am going to recognize myself first. I have one question.

Dr. Hayes, I would like to be clear, myself, and be clear for the
record. As you well know of course there are right now four
supercomputing centers funded in the NSF program.

What I would like to know is. Does your Task Force's report spe-
cifically recommend more such centers, less such centers? Numeri-
cally speaking. I know you made several recommendations on how
to network and so forth, but in terms of the number of centers, is
there anything in the report that addresses that?

Dr. Hayes. As the Task Force looked at the overall program, the
advice that we had had fi-om the Foundation and our own sense of
what the budget environment was going to be for the next five to
ten years, suggested that in order to have a balanced program, that
it may be necessary to downsize the total number of partnership
sites.

I would also mention that the Task Force was also aware that
an important part of a growing component of the program is the
Metacenter Regional Affiliates.

In our recommendations about the partnership centers, we recog-
nized that these Metacenter Regional Affiliates could and should be
brought into the program as a natural evolution of the centers pro-
gram.

So within the Task Force there was an expectation that in order
to fund the balanced program that was envisioned, that it may re-
grettably be necessary to reduce the number of centers.

Mr. SCHIFF. But at the same time, you are proposing the origina-
tion or inclusion of other centers — I'm sorry? use the term? —

Dr. Hayes. Metacenter Regional Affiliates.

Mr. SCHIFF. Right.

Dr. Hayes. Yes, that presently are a part of the program, and
they are loosely coupled to the existing four centers.

Mr. SCHIFF. Right.

Dr. Hayes. I think one of the important things to appreciate is
that between the time of the Branscomb Committee Report and
when the Task Force was called together, there were three impor-
tant changes.

One had to do with the expectations with respect to the budget.
The Branscomb Committee, which I was a member of, was working
in an environment in which there was some hope and expectation
there would be significant growth in the budget for high-perform-
ance computing.

So the possibility that we would have to downsize the number of
centers was really not addressed in a formal way as being driven
by constraints on the overall budget.

The other is a full appreciation of the importance of having a bal-
ance within the p3rramid; that the mid-range systems, because of
important developments in the technology, were becoming increas-
ingly important.

So the possibility of buying from major commercial vendors mid-
range systems that were upwardly compatible by increasing the

total number of nodes on the systems really became a reality in
that very short period of time. And the importance of building on
that development in the technology was very much before us as the
Task Force was formed.

Our sense is that it was important to recognize that there were
a number of these mid-range centers that could be coupled more
effectively into the program, and there will be efficiency and cre-
ativity that would come from that formal coupling rather than
loose coupling as exists today.

Mr. SCHIFF. Thank you, Dr. Hayes.

Congressman Cramer?

Mr. Cramer. Thank you. I apologize for having to be out of the
room for a few minutes.

Welcome, both of you, and thank you for your testimony.

Dr. Hayes, are there any significant aspects of the recommenda-
tions of your committee that are not being followed by NSF in re-
structuring the supercomputer centers program?

Dr. Hayes. I am not aware of any major elements that are not
reflected in the program announcement.

There are other issues with respect to the high-performance com-
puting that we expect that the Foundation will follow through on,
such as taking the leadership with other agencies in high-perform-
ance computing and continuing to look at options to fine-tune the
review process for allocation at the centers, but those are not mani-
fest specifically in the program announcement.

Mr. Cramer. Dr. Young, the solicitation for the new centers pro-
gram requires submission of a preproposal

Dr. Young. Yes.

Mr. Cramer. So you are going to have a preproposal, if I can say
it right, and a final proposal as well. Will there be overlap between
reviewers of those two sets of proposals?

Dr. Young. I would certainly expect overlap. They may not be
exactly the same group, but I would expect very significant overlap,
yes.

Mr. Cramer. So you would expect that that would be monitored
so that we don't have problems as a consequence of trying to form
what you do around one, and then there not being continuity be-
tween the final proposal and the preproposal reviewers?

Dr. Young. The intent of the preproposal stage is to allow
preproposers to come in and get advice about the quality of their
proposals; to get input; and to not use the preproposal rankings as
a determinant factor in what the final will be, but rather as a
mechanism for giving advice to people coming in.

Mr. Cramer. And I want to say, I like that idea. That is what
I had hoped I would hear, because I think that is a good way to
explore it through that process that work with and try to cooperate
and encourage that.

Thank you, Mr. Chairman. That's all I have.

Mr. SCHIFF. Mr. Ehlers, you are recognized for five minutes.

Mr. Ehlers. Thank you, Mr. Chairman.

My question is somewhat tangential and I hope you do not think
this reflects on your report, but I think your testimony and the re-
port speak for themselves.

We have a particular problem in the Federal Government in
terms of the purchase and procurement of computers. The procure-
ment process itself is extremely complex because of the paranoia of
the public over the years that someone might be given favored con-
tracts, or some such, and it has reached the point where, when the
Federal Government wishes to purchase a new computer using the
process that exists there is an absolute guarantee that the com-
puter will be obsolete by the date it is delivered.

I have two questions based on that.

Number one. Is that a problem in any of the centers? Are you
in any way bound by any of the procurement requirements, either
directly or indirectly, that might be imposed through NSF?

And secondly. Do you have any advice to give us, particularly
since I am looking at ways of restructuring the federal procurement
policies vis-a-vis computers, do you have any advice on how I could
proceed or how we could proceed in order to eliminate the problem
we have now?

I would welcome comments from either one of you.

Dr. Young. Well I will take that first, and I may turn to Bob
Borchers if he has backup comments he wants to maike.

First of all, we do cooperative agreements with vendors. We are
on the leading edge, so we are not bound by the federal procure-
ment process in acquiring the leading edge computers here.

What we typically do is partner with vendors to bring in very
early versions of machines that are often very interested in coop-
eratively working with us on software development for scientific
applications because that is not the primary drive that is driving
them for their mass market.

So we have a mutual interest in doing software development and
early development on machines that enables us to get very com-
petitive rates in what we do in leasing.

So the flat answer to your question is: we simply are not bound
by the procurement process, and it works very well to partner with
industry to bring these machines in for early testing and develop-
ment.

With regard to the second question. No, I do not have advice on
what to do about the procurement problems. I recognize them.

Mr. Ehlers. Thank you.

Dr. Hayes?

Dr. Hayes. Yes. I would echo the comment about the efficiency
of the system. I think that many center directors that are in the
Federal Government just marvel at what the center directors — at
what our NSF Supercomputing Centers have been able to do in
terms of getting leading edge equipment on site and doing it in a
very short time frame.

The key point that Paul mentioned that I think it is important
to recognize is the extent of partnerships between the centers and
the vendors. I think that has been a key element in the success of
the program right from the beginning.

I think one of the new elements in all of this that is important
to keep in mind is that the recapitalization cycle for the new hard-
ware is getting shorter. One of the factors that went into our con-
sideration as to how much budget would be required to support a
center was part of the experience from the history of the program

that if you run with long recapitahzation cycles on leading edge
technology, that it leads to problems within the program.

So that the center directors themselves may have some addi-
tional insights on this particular question, but I think as long as
the resources are there in a timely way and importantly they do
not make these very long-term commitments that prevent them
from recapitalization on an appropriate time scale, which possibly
in this environment is three or four years, why you can see the
kind of turnover in a timely way that is very important to the suc-
cess of this program.

Mr. Ehlers. And do you have any advice for us?

Dr. Hayes. No, I would not hazard advice.

[Laughter.]

Mr. Ehlers. We may need a Hayes' test for a federal computer
system.

Dr. Young, did you have anjrthing to add?

Dr. Young. No.

Mr. Ehlers. Thank you, very much.

Mr. SCHIFF. Mr. Doyle?

Mr. Doyle. Thank you, Mr. Chairman.

Let me see if I have got this right. Now the principal role of the
new centers program is to provide access to high-end computing in-
frastructure for the academic, science and engineering community?

Is that correct?

Dr. Hayes. That is correct.

Mr. Doyle. And, Mr. Hayes, your report indicated that the
supercomputers at the existing center are oversubscribed by a fac-
tor of at least two, and the demand is growing.

Most of us here understand that little or no funds are available
for the restructured program which must fund many regional cen-
ters and research consortia in addition to each leading edge
supercomputer site, however many there are, one or two; and, fi-
nally, ARPA is not likely to provide support to the new centers for
hardware upgrades as has been the case for the current centers.

Therefore, since the new program will be increased significantly
in scope without additional funding, why do you believe the new
program will adequately support high-end computer users?

Dr. Young. I don't believe it will adequately support high-end
computer users. If you look across the Foundation as a whole, per-
haps one in four of the proposals for research that come into the
Foundation that are good proposals are in fact funded. So these are
tight times and Chairman Schiff reminded us of that at the begin-
ning. We are making tough choices.

Dr. Hayes. I would say that part of the perspective that the Task
Force brought to the table was the realization that there is a very
large amount of important computational research that can be done
on mid-range systems and work stations.

The view of the Centers Program is not to provide support for
the vast majority of computational studies. The NRC report on
the — the Brook-Sutherland Report talked about the Centers Pro-
gram and the highest end computing in the context of a time ma-
chine; that the work that is being done on these leading edge sys-
tems that are very expensive, that kind of work is basically trail
blazing work developing algorithms and methodology, testing new

physics and chemistry and mathematics, that will be used widely
on mid-range machines in the next five to ten years.

Why? Because the technology capability, the underlying tech-
nology capability is doubling every 18 to 24 months. So today's
work stations basically are equivalent to the supercomputers that
were used in the early stage of the program.

So from a national perspective, I think the most important aspect
of this program is to provide a cadre of scientists in many fields
with access to the highest end. Not that it is going to be the total
effort in the Nation, facilitating the total effort, in high-perform-
ance computing, but it is going to be providing the cadre of the best
and brightest students that are going to be doing trailblazing re-
search, but there is still going to be the vast majority of the work
that is going to be done on mid-range machines and on work sta-
tions.

That was one of the important points that was made in the
Branscomb Committee Report, and I think that is being borne out
by what is happening in many fields today.

Mr. Doyle. I have one more question.

I am just curious, and a lot of us are just trying to understand,
you know, what the driving forces are. We hear all the reports on
the four centers. They have all been glowing reports, and a lot of
us wonder why can we not just make modifications at the four ex-
isting centers driving this movement to make these significant
changes?

Would it be fair to say that, as you look at the budget — you
know, we all understand the budget climate that we are in right
now. I guess what I am trjdng to understand is. Is it dollars that
are driving these decisions to look at maybe going from these four
centers to maybe one, maybe two?

Why could we not do this by modifying what goes on at the four
existing centers and spare all the expense of a recompete?

What is driving this?

Dr. Young. I will take that, or attempt to take it.

It is certainly not being just driven by dollars, though that cer-
tainly drives part of it. If you have a constant budget and you allow
for inflation, it gets hard to do as much as you have done in the
past on a fixed budget.

But as Ed pointed out, the technology is also driving the struc-
ture of the new program. There are two parts to that technology.

One is the increasing viability of mid-level parallel machines
which scale up to the high end, as he mentioned. That enables new
regional and state centers, for example, or groups at universities to
partner with centers in a sjmergistic fashion to make use of the
technology in a more effective fashion.

We think that this new program will use that technology and a
corresponding increase in high-band width connections which will
enable these centers to partner together in a more effective way to
restructure the program so that, although we will have fewer high-
end leading centers, we will have the totality of more full partners
in the program and we think we will get greater efficiency with
that.

Mr. Doyle. Dr. Hayes?

Dr. Hayes. I think the concept that we are recommending —
namely, the partnership centers — is right, irrespective of whether
the number is two, three, four, or five. So let me be clear about
that.

Just two amplifying insights here:

One is the realization that the very best people in computer
science and in computational mathematics and in computational
engineering generally are not all located at one place. So that the
partnership centers, as we have seen with the Metacenter Regional
Affiliates, play a very important role in the training, and develop-
ment, and the inclusion of a much broader cross-section of talented
people within the concept of the program.

So the model facilitated by the technology we believe to be the
correct model; and that the effort to coordinate the development
and the acquisitions and the allocation of times is well worth the
effort from a national perspective.

The other point that is of course present here is that the tech-
nology is changing quite rapidly. And that if you do not keep these
leading edge sites one to two orders of magnitude ahead of what
you can do with a work station, the attractiveness and the con-
tributions that will be forthcoming from these centers will be sig-
nificantly reduced.

So you have these two things that are tugging at each other and
are important changes in the computing environment from what it
was at the time of the initiation of the Centers Program.

Mr. Doyle. Thank you, Mr. Chairman.

Mr. Ehlers [presiding] . Thank you.

I think Congressman Gutknecht left — oh, no, I am sorry. There
he is leaning back.

Mr. Gutknecht. I am here, but I would yield to somebody else
who has questions.

Mr. Ehlers. Congressman Gutknecht yields to Representative
Morella who has recently joined us.

Mrs. Morella?

Mrs. Morella. I thank the gentleman for yielding to me, and I
thank you, Mr. Chairman.

With the budget released today, what is NSF's budget for the
Supercomputer Centers for Fiscal Year 1997?

Dr. Young. For Fiscal Year 1997 it will be essentially the same
as Fiscal Year 1996, which is roughly — and I may be wrong by a
little here — $67 million.

Mrs. Morella. Sort of level funding

Dr. Young. Level funding.

Mrs. Morella. (continuing) — for the 18 months. And I guess
that ties into the question that was asked of the four directors that
will be succeeding you on this whole concept of is it collaboration,
or is it competition? And does that then set out turfdoms? And
what do you do about that?

Is there anything further you would like to add?

Dr. Young. When they testify, the directors themselves may
want to testify about that.

I think one of the big successes of the Supercomputer Center
Program over the past 10 years has been the degree to which com-
peting programs have learned to work in a cooperative fashion.

I think there has been a great deal of collaboration and very
fruitful interaction among the existing Supercomputer Centers.

I think one model of the developing program is that we will have
partnerships that extend beyond the current leading edge sites. To
some extent that is modeled on cooperation that already exists.

It is unquestionably true that, in the context of a recompetition
that, it does encourage each of the current centers that may be
coming in in the recompetition to put forward the best proposals.

At the same time, it is NSF's management's intent to continue
cooperation among leading edge sites and partnerships as they
come in.

But you are right. It presents new challenges for our manage-
ment, and we intend to follow through on that, and we intend to
monitor cooperation and put in place procedures which will encour-
age cooperation in the future.

Mrs. MORELLA. So with the procedures you have in mind and the
concept, you think this is going to work out well.

Dr. Hayes, do you want to add to that in any way?

Dr. Hayes. Yes. I would add two perspectives which expand on
the points that Paul has made.

One is that competition, if it is done in an appropriate environ-
ment, can bring out creative new ideas and it has many of the posi-
tive aspects. So the challenge — and I think, adding to Paul's
point — the challenge for NSF is to figure out how you keep the
playing field level and permit an appropriate level of competition
to bring out the creative juices, if you will, as part of this program.

And I think there are sort of two levels of partnerships that they
have to pay attention to.

One is the partnerships within the partnership centers. Then,
once the competition is over — and the Task Force was very strong
in recommending that two centers is infinitely better than one;
okay? so our expectation is that when there are two or more win-
ners, there is still an important issue of partnerships among the
centers that win out.

NSF has a good record with the present centers in encouraging
and facilitating that kind of collaboration and partnership, as well.

Dr. Young. May I come back to that

Mr. Ehlers. Yes, you may.

Dr. Young, (continuing) — in addressing partly that question and
the earlier question about why take a successful program and
recompete it, which I think was implicit.

The current program has been very successful, but part of NSF's
standard management is to take peer review of the very best ideas
and always put things out for recompetition after a certain amount
of time. We think that is very important.

We think that with the current solicitation we are going to get
a lot of new ideas, and very creative ideas, on how to structure this
program.

Ed mentioned earlier in his testimony that our solicitation allows
for a great deal of creativity in how these programs come together,
and I think a little competition and a chance for people to come in
with new ideas about the appropriate structure of the program is
going to lead to a very exciting new program, and I think that is
going to work well.

26-018 0-96

Mrs. MORELLA. I just wondered, if I have another moment, sim-
ply because from what I have read — and I am not expert on this —
that there have been high marks given by at least two review pan-
els of the current system and why modifications can't be made
without the additional expense that would be incurred by the
recompetition.

Do you have any comments? Do you wonder, too?

Dr. Young. I think I actually tried to answer that in the earlier
response. I believe that we will get a significant restructuring and
better use of existing technologies through the competition.

There is a price to doing that, but I think one of the things that
is true about Foundation policies and procedures generally is that
we find open competition — peer-reviewed — a useful way to bring
out new ideas, and that is a process we are going through. It is
often painful and it is not without cost, but we do believe that that
is the best way to bring new ideas forward, and I think this new
solicitation will do that.

Mrs. MORELLA. Thank you. Dr. Young.

Mr. Ehlers. The gentlelad/s time has expired, but if you have
a brief comment you can add to that.

Dr. Hayes. Just to follow up on the competing visions for the fu-
ture in terms of how one is going to deliver the highest perform-
ance computing, I think it is important to recognize that there have
been significant developments in the networking capability, and
they are projected for the future.

People have different and legitimate differences of opinion about
what the future holds. So one of the things that is going to be
worked out as part of this competition is just what the balance is
between how people are going to be doing the highest end comput-
ing and the extent to which they will be doing it in a distributed
mode.

Mrs. MoRELLA. Thank you, very much.

Mr. Ehlers. We have been joined by Congressman Volkmer. Do
you have any questions, Mr. Volkmer?

Mr. Volkmer. Yes.

I want to first apologize for not being here for the full testimony,
but I reviewed some of your statements and I was just discussing
it with the gentleman from Pennsylvania and I would like to con-
tinue along a similar line because I just do not quite understand
the reasoning for the recompetition. But from what he has told me,
perhaps there is a need for it if you want to try something dif-
ferent.

But do you, when you envision what is going to come from the
new competition, do you have any ideas of what you are looking for
to come forward?

Dr. Young. Yes.

Mr. Volkmer. What are those ideas?

Dr. Young. We believe that networking technology and the ad-
vent of scalable mid-level parallel machines that scale up to the
high end enable us to restructure a program in which we can get
a broader distribution of the technology across the country, more
participating centers, and better integration of mid-level systems
and better use of high-speed networking connections into a pro-

gram that is less centralized and broader than the current pro-
gram.

Mr. VOLKMER. All right. So what you envision is, you would still
have two, three, four, or five main centers?

Is that correct?

Dr. Young. That is correct.

Mr. VoLKMER. But they would be friendly users?

Dr. Young. Yes.

Mr. VOLKMER. So that other universities and other research cen-
ters would be able to utilize the information that they have devel-
oped, or what?

Dr. Young. Not only utilize the information they have developed,
but partner in the development of new software to partner in the
development of new algorithms and work in a fashion that is coop-
erative with them using the fact that they have similar models of
machines and similar software developments.

Ed spoke earlier of the fact that the expertise in computing in
the country is not located at just a few institutions. We believe that
we can get a better partnering of the expertise that is around the
country by a program that makes use of these new technologies.

Mr. VOLKMER. And the ultimate aim within four to five to six
years is what?

Dr. Young. The ultimate aim is to have the ability to do com-
putational science and engineering distributed even more broadly
across the country and have a more powerful and better program
for the country as a whole.

Mr. VOLKMER. Would we end up then with a perhaps new and
different software

Dr. Young. We will certainly have new and better software, and
we think that the restructured program will actually aid in that de-
velopment of software both for the underlying technology and for
the specific technology for computational science and engineering.

Mr. VOLKMER. And this could be transferred to private industry
and private business?

Dr. Young. We believe that much of the work that goes on in the
Foundation in this area ultimately finds application in business, for
sure. The

Mr. VOLKMER. All right. Fine.

Now do you envision that those who will be entering the competi-
tion — or is the competition still open?

Dr. Young. Pardon?

Mr. VoLKMER. Is the recompetition still open?

Dr. Young. Yes, it is. The deadline for preproposals is April
15th.

Mr. VOLKMER. All right.

Do you envision — about how many will be in that competition?

Dr. Young. I would guess on the order of four to half a dozen,
maybe more.

Mr. VOLKMER. Four, five, six, seven?

Dr. Young. Yes.

Mr. VOLKMER. Mostly the same centers that are out there right
now?

Dr. Young. That is the current indication, I believe.

Mr. VoLKMER. So basically we have a recompetition with, to me,
a goal of changing what is being done right now at those same cen-
ters, it looks like.

Dr. Young. It will change what is going on at those centers, and
it will change what is going on in other centers around the country.

Mr. VOLKMER. Well the other centers will change as a result of
what these centers do. Is that correct?

Dr. Young. I think both will change.

Ed?

Dr. Hayes. I think I need to draw a little cartoon of what this
Partnership Center looks like.

Mr. VoLKMER. All right. That might help.

Dr. Hayes. A Partnership Center has a leading edge site

Mr. VOLKMER. All right.

Dr. Hayes, (continuing) — which has a high-end balanced system.
But as part of the partnership there may be four to five — the exact
numbers will depend on the proposals of course — but of partner-
ships where there is human resource capability; there are existing
networks; and there will be hardware and software capability.

The center is the combination of the leading edge sites, as well
as the partnership sites.

Mr. VoLKMER. In other words

Dr. Hayes. It has been recognized since the beginning of high-
performance computing centers that a good deal of the use of these
high-end systems in fact is used by people who have major research
problems, but on a job-by-job basis, particularly in the development
of the software, may not require access to the highest end of the
performance spectrum.

So within this new Partnership Center there will be the possibil-
ity, particularly in the early development stages, to allocate time,
resources, and training on the mid-range machine.

Why?

Because now it is technically possible today to — for instance Ohio
State is affiliated with the Ohio Supercomputing Center, which
serves all the universities within Ohio, which is a Metacenter Re-
gional Affiliate, which is affiliated with the Pittsburgh Center, but
it is very loosely coupled.

It is not possible for the Allocations Committee to say yes to a
researcher, we think you have a good idea but we want more quan-
titative data. We want you to go to a Partnership Center like the
Ohio Supercomputing Center and demonstrate to us that you are
technically ready to make effective use of the high-end resource.

When you are ready, then you will be able to compete with oth-
ers to have that access.

I think in a sense it is akin to getting preliminary data for a
large experiment. This is viewed as a more cost-effective method of
doing that because now it is the whole center that is being man-
aged.

The other point has to do with software productivity. I think es-
sentially everyone who has been looking at the performance of the
centers program over the years has recognized the important
science that has been done. But the advances in terms of software
productivity have not been in parallel — have not made progress in
parallel with the underlying technology that is being used.

One of the hopes is that by bringing in a broader set of partners,
computer scientists, mathematicians, that can work on early ver-
sions of machines which may at some point be the leading edge
systems, that we will get significant advances upon which everyone
will benefit in terms of software productivity and, importantly, in
learning how to use distributed systems over the network.

I think the issue is on what sort of time scale are we going to
find that the highest performance computers in fact are assembled
by using computers that are distributed across the network.

This is a very important issue for industry, as well as for re-
search in this country.

Mr. VOLKMER. Mr. Chairman

Mr. Ehlers. The gentleman's time has expired.

Mr. VOLKMER. May I ask one more question?

Mr. Ehlers. If it is a very, very brief one.

Mr. VOLKMER. It is.

You have the primary center, what I call the primary center, and
an ancillary leading edge.

Dr. Hayes. Yes.

Mr. VoLKMER. Now those that join with that leading center,
would that be only university and academia? Or will that include
industry and private business at the research centers?

Dr. Hayes. We include industry, national laboratories-I think
there

Mr. VOLKMER. Whoever they-

Dr. Hayes. It is specifically stated in the program

Mr. VOLKMER. Whoever they can get to sign up, in other words?

Dr. Young. That is right. I could read from the program solicita-
tion. You probably do not want that, but just a subset of those list-
ed. It is not an exclusive list, but it includes universities, and in-
cluding research groups within universities, centers of various
forms, research and educational consortia organizations and
groups, regional and state supported high performance computing
centers, private-sector organizations, and national labs.

That list is not intended to be exclusive.

Mr. VOLKMER. Thank you.

Mr. Ehlers. Thank you.

We have been joined by Congressman Boehlert. Congressman, I
would like to ask you to take the Chair, and then you may have
your turn to ask questions, as well, because I will have

Mr. Boehlert. What a deal that is.

[Laughter.]

Mr. Ehlers. Because I have to leave for the Floor to defend a
bill.

Mr. Boehlert [presiding]. What a deal this is. I get the Chair
and the opportunity to question.

I am not quite certain I am clear in my own mind why we are
going in the direction we are going, because what I see is ten years
of success and we have got four outstanding centers doing a lot of
things you are talking about — their extensive outreach program,
their partnering, bringing in the private sector — I mean, you do not
have just four little islands unto themselves scattered around the
country.

It seems to me you are talking about reducing the number of cen-
ters at a time when I would think we would be expanding what we
are doing at the centers and encouraging them to keep up the good
work.

I am not quite sure I am clear why there seems to be the drive
toward a reduction in the number of centers after a decade of suc-
cess.

And once again, I seem to think — or that is my impression at
least — that the centers are all individually doing exactly what you
are talking about what you envision as a result of this competition,
the extensive outreach; the partnering; the bringing in the private
sector.

Enlighten me a little bit, if you will. Doctor?

Dr. Young. Shall I try that first and then let you come?

Dr. Hayes. Yes.

Dr. Young. I think your point is well taken. It may be a case
of success breeding success.

But I think it is also true that the technologies permit an accel-
eration of that process and a deepening of that process.

So I think that you will see more partnering and more outreach
with the program as it is structured.

As Ed mentioned earlier, and I think before you came in, the
structure that is envisioned here is a structure that makes sense
independently of the actual number of centers and partnerships. So
we think that this intent to make better use of networking and
scalable parallel computing is something that the centers, as you
point out, have been doing, and we think it is a natural evolution
of the program in that direction.

At the same time, we do have constant budgets, tight budgets,
and we still have inflation. There is a limited amount we can do
with tight budgets.

Ed mentioned that one of the things we need to do at the leading
edge sites is to keep them one or two orders of magnitude, ten to
one hundred times faster, ten to one hundred times more storage
power, just the technology needs to be a couple of places ahead.
And that is expensive and there are a limited number of places we
can do that.

We think we are going to get a more efficient program as we ex-
pand some of the things the centers have been doing very well.

Mr. BOEHLERT. Dr. Hayes, do you wish to add anything to that?

Dr. Hayes. Yes. I think one of the futures that the Task Force
was very concerned about is the scenario where we get four Part-
nership Centers, where the Metacenter Regional Affiliates that we
currently have, that that part of the program is folded into the
broader program, so you get four Partnership Centers.

Mr. Boehlert. Which is in effect what we have right now.

Dr. Hayes. Well I will come back to that as a detail. That is not
quite right. We have four leading edge sites right now, and we
have loosely coupled Metacenter Regional Affiliates. That is what
we presently have.

The concern is that, with the changing philosophy at ARPA,
DARPA now, that if these centers came into being and the NSF
budget did not grow at a rate that was significantly above inflation
for this program, that you would not be able to keep up with the

recapitalization cycle that would be necessary to keep the leading
edge sites at a level that would be sufficiently interesting to draw
the very best researchers to the centers.

You may want to explore this with the center directors, but the
impression that many of us have is that without the DARPA
money to help NSF move into the massively parallel arena over the
last five years of the program, why many of the centers would have
been stuck at the vector computing stage, because many had long-
term leases or payments on existing centers. And without an infu-
sion or a recapitalization, you could find that what appeared to be
a good investment and positioned the center for a period of two or
three years basically put a Partnership Center in the position that
they were not really capable of supporting the most interesting
science, and that that would basically put you on the kind of spiral
where the scientific community, the people who actually do the
science, would say, well, by buying a mid-range system at signifi-
cantly lower cost, they can do research which is competitive with
what you could do at a Partnership Center with six- or seven-year-
old technology.

So the concern about "sizing" the program that was very much
on our mind is that the program needs to be sized so that it can
be recapitalized on a reasonable time scale.

People debate about whether that is three or four years, but I
think there are a few that would argue that five to seven years is
the right recapitalization schedule.

So when you think about within the kind of budget environment
they are in, and the kind of funding that is available for the Cen-
ters Program, recapitalization on a shorter time scale, and also
bringing in the partnership aspect and being able to do realistic
and interesting experiments with distributed computing, the
present centers do not have that to the same extent that is envi-
sioned in our recommendations.

The current centers are doing many very important things, and
I do not want to in any way be critical of what they are doing
under the present program — and I think the Task Force wasn't.

But underlying all of this is the fact that, you know, we are
Blessed with the fact that technology is changing very rapidly, and
in thinking about what the program may look like over a five- to
ten-year period we need to have the flexibility within the program
so that it can recapitalize and keep up with the technology.

Mr. BOEHLERT. It seems to me you are making a persuasive case,
or fighting like hell for more bucks for the program, rather than
downsizing the program or restructuring the program.

Is it your interpretation of budget realities that is forcing this
new line of thinking?

Dr. Hayes. Let me be clear about two points.

There is nothing about the partnership concept that is related to
the budget. I think the Task Force would be unanimous that the
partnership aspect, where you have a leading edge site that is
tightly coupled, not loosely coupled but tightly coupled to partner-
ship sites, is the right model for the future.

Mr. BoEHLERT. But is that not what is evolving right now under
the present configuration?

100

Dr. Hayes. There — I would — you will have to ask the center di-
rectors their view on that. I happen to be familiar with one
Metacenter Regional Affiliate. There are a number of aspects that
we would expect to be present in a Partnership Center in terms of
tight coupling and management and planning, which at least do
not appear to be present in that particular Metacenter Regional Af-
filiate.

The coupling is much looser. The planning is looser. And the
ability to — in fact there is no ability for an Allocations Committee
to allocate resources at the Metacenter Regional Affiliate that I am
familiar with.

So if a proposal comes into NSF from New York State, for in-
stance, and says, look, they want to carry out a particular project,
and the Allocations Committee says, well, what you need is a mid-
range kind of machine, and that machine is available at the Ohio
Supercomputing Center, there is no way that they can allocate
time on that particular resource.

So what happens is, either the particular project does not get any
allocation at all because there is uncertainty about whether it is
technically ready, you see, or the decision is made, sometimes pre-
maturely, to allocate resources on a very high-end system that is
costly to iDasically demonstrate that they are technically ready.

It seems to me that leads to inefficiencies in the program.

Mr. BOEHLERT. Congressman Doyle?

Mr. Doyle. Thank you.

I just want to get one thing — ^We have four centers right now.
And, Mr. Boehlert, when you were talking about what is going on,
I think the reality is here, though, that under what is being pro-
posed here, what we are going to end up with is one, maybe two
leading edge centers.

Is anybody disputing that?

Do you see more than two leading edge centers, given the budget
constraints we are under right now?

Dr. Young. I think it is plausible that we will end up with be-
tween one and three.

Mr. Doyle. Three now?

Mr. Brown. Between one and three.

[Laughter.]

Mr. Doyle. Yes. "Between one and three" is two, yes.

[Laughter.]

Mr. Doyle. And the other observation is, I am just sitting here
looking and thinking that a lot of the changes you are talking
about, while they are going to open up, you know, more to mid-
range users, it seems to me it is going to be at the expense of the
high-end users; and the high- end users are oversubscribed as we
speak.

I am just wondering, you know, it gets back to what Mr. Boehlert
was sajdng, too. Maybe we need some people to sit up here and talk
about the need for some additional funding for what has been a
very successful program at these four regional centers.

It just seems that what is going to happen here is we are going
to open up the process for some at the expense of high-end users
that are already over-subscribed and end up with one, maybe two.

101

leading edge facilities here instead of our four regional centers.
That is just what it appears to me to be, anyway.

I would be interested to hear your comments.

Dr. Hayes. I think the point I made in my prepared statement
about the process that NSF has is that I think the Task Force felt
comfortable with the concept of the Partnership Centers, as I said.

But the subsequent steps that took place within the Foundation
were that basically the Director had to take a plan to the National
Science Board, and the Director and the National Science Board in
the context of the whole NSF program, the whole budget, had to
make a judgment on this as to what was a reasonable balance be-
tween high-performance computing and the rest of the NSF pro-
gram.

The Task Force certainly heard loud and clear that many of the
people that are using the existing centers in fact depend upon sup-
port from NSF through disciplinary programs and other programs
for their support.

So if you, again, look at things in that kind of perspective, some-
one has to look at the total NSF portfolio within what is a reason-
able expectation of the budget. I think this Committee looking at
this is very important can make a judgment that we want to keep
the whole organic whole healthy.

So if you have somebody like Jerry Ostriker, who Paul men-
tioned, or another one of the major users basically that loses fund-
ing for their overall program which is not, you know, exclusively
a high- performance computation program, if you visit any one of
these laboratories, including Ostriker's, he has a lot of mid-range
computing capability in his laboratory that is coupled appro-
priately.

Not everyone is so fortunate. I use Jerry Ostriker as an example
because I think it is quite unlikely that in the budget reduction he
loses his support, but I think that that is one of the issues as to
what gives us the best overall plan.

The sense of the Task Force was that you really need to have a
minimum of two Partnership Centers. And if the budget can afford
additional centers or, importantly, if there is additional significant
cost sharing that comes from vendors and from the partners and
the leading edge sites and from the states, that in fact it may be
possible to do more than two. And if the NSF budget would sup-
port, with the recapitalization that I mentioned earlier, more than
the minimum of two that we were strongly pushing for, then within
the concept of the partnership I think there will be quite a comfort
level and an enthusiasm for doing that. But only if it meets the re-
capitalization and the balance aspects that I was talking about.

Mr. BOEHLERT. Thank you very much, and thank both of you
very much.

We will now move to the next panel consisting of Dr. Malvin
Kalos, Director of the Cornell Theory Center; Dr. Larry Smarr, Di-
rector of the National Center for Supercomputing Applications; Dr.
Douglas Pewitt, Acting Director, San Diego Supercomputing Cen-
ter; and Dr. Ralph Roskies, a Scientific Director for the Pittsburgh
Supercomputing Center.

Gentlemen, your statements will appear in the record in their en-
tirety. I would ask in the interests of time you give us as much as

102

we need to know so that we can ask intelligent questions and un-
derstand the issue, but please do not be too lengthy.

I would ask also that we follow the order of introduction with the
privilege of leading off going to a fellow New Yorker, Dr. Kalos, as
soon as you are comfortably seated.

Dr. Kalos. Thank you

Mr. BOEHLERT. And would you pull the mike a little bit closer?

STATEMENT OF DR. MALVIN H. KALOS, DIRECTOR, CORNELL
THEORY CENTER, AND PROFESSOR OF PHYSICS, CORNELL
UNIVERSITY, ITHACA, NEW YORK

Dr. Kalos. Mr. Chairman, it is my privilege to be invited to com-
ment on the NSF's new program of Partnerships for Advanced
Computational Infrastructure.

I am Malvin Kalos, the Director of the Cornell Theory Center,
one of the four current NSF Supercomputer Centers. I am a Profes-
sor of Physics at Cornell University. I have been a computational
scientist for more than 40 years, and at present am the Vice Chair-
man of the Division of Computational Physics of the American
Physical Society.

Our Center was founded with the premise that computation has
a profound effect upon all the sciences and engineering.

From the beginning, our Center took a leadership role in scalably
parallel computing. Now computational science is undergoing a rev-
olution with the advent of practical scalable computing.

The 512-processor SP at Cornell is one of the most powerful com-
puting environments available today. We have seen leading re-
searchers in our community cross new thresholds of capability to
do qualitatively new science.

In the end, a program like this and our Center can only be
judged by the quantity of important scientific knowledge and un-
derstanding which could not have been accomplished without it.

I would like to call attention to a few special examples that illus-
trate the scope and influence of this program.

My first example is one referred to by Dr. Young in the impor-
tance of computation for molecular biology:

An international team of scientists investigated the rapid activity
of acetylcholinesterase which breaks down the neurotransmitter ac-
etylcholine.

Its three-dimensional crystal structure was solved a few years
ago. It seemed inconsistent with the rapid djmamics of the enzyme.

Using computing resources at all four of the NSF Centers, the re-
searchers discovered that there was a back door to the molecule
that explains the rate.

Since inhibitors of acetylcholinesterase are important medica-
tions for diseases such as myasthenia gravis, glaucoma, and Alz-
heimer's disease, this new insight may lead to more effective drugs
to fight these diseases.

A second example is in astrophysics:

Using both the NSF-supported Arecibo radiotelescope for obser-
vation and the Cornell Theory Center's computational facility for
intensive analysis and modeling, Alexander Wolszczan of Perm
State University succeeded in identifying the first planets outside
of our solar system. Wolszczan was recently awarded the pres-

103

tigious Beatrice M. Tinsley Prize of the American Astronomical So-
ciety for this discovery.

John Dawson and his research group at the University of Cahfor-
nia at Los Angeles studies turbulence in complex plasmas. The
power of the Cornell SP permit his group to support simulations
that are an order of magnitude improvement over previous ma-
chines in spatial resolution, which has allowed them for the first
time to simulate successfully the parameters of a large fusion de-
vice.

Here, as elsewhere, computation as an aid to understanding and
as an aid to predesign of costly experiments is an important way
to use limited budgets in an optimum way.

At the Massachusetts Institute of technology, Edmund
Bertschinger conducts Grand Challenge research aimed at under-
standing the formation of galaxies in the universe. The power and
memory of computers such as the SP permit him and the rest of
the Grand Challenge Team to study models large enough to cap-
ture these large-scale dynamical effects. This, too, is work that
spans the NSF Centers Program.

I would like to echo Dr. Hayes' comments about the relationship
with industry.

We believe that we have also made a singularly important con-
tribution to the national scientific effort by the depth and quality
of our partnership with IBM.

Through this partnership, we rekindled IBM's interest in high-
end technical computing and thereby helped to bring their superb
technology to the service of science and engineering.

As is often the case, the technical computing world has led the
way in bringing out new technologies for the broader commercial
arena.

We believe that IBM's success will help assure a future for com-
putational science.

Computation is a unique tool that permits quantitative connec-
tions among different disciplines. Every one of the large problems
that confront our society, and to whose solutions we expect science
to contribute, is in face a multi-disciplinary problem.

Issues of the environment and medicine, to cite only two, involve
many sciences, chemistry, physics, engineering, fluid flow, biology,
materials. Bringing the knowledge from these fields together to
make quantitative predictions — for example, about the effects of
some technological proposal — would be utterly impossible without
the use of high-performance computational modeling.

It is the indispensable natural language of quantitative multi-
disciplinary research.

Computational science is now an essential tool in experimental
science. The most advanced scientific instruments — optical and ra-
dial telescopes, particle accelerators, and computers themselves —
are designed, studied, optimized, and verified with computer sim-
ulation.

Data collection is usually automated and the reduction to com-
prehensible data sets or conceptual models may involve enormous
computations as in the case of Alexander Wolszczan.

104

Now the entire scientific and engineering community of the coun-
try has the opportunity to exploit the new tools of computational
science.

We share NSF's vision that education of tomorrow's young sci-
entists includes bringing them into the process of exploiting the
Nation's most advanced tools, including the computational re-
sources of the NSF centers. They will carry the message to the rest
of our society and to the future.

The supercomputing community now finds itself at a major cross-
roads. To exploit these new machines, the scalably parallel ma-
chines, a major retooling of software and algorithms must take
place.

In response to the new NSF program, but also to assure that re-
cent advances are sustained and extended, and that new challenges
can be met. The Cornell Center is creating a National Alliance for
Advanced Computational Infrastructure. This will comprise a new
network of cooperating organizations around the Nation that will
bring their varied talents and their resources to the Alliance, along
with the world-class supercomputing infrastructure of the Theory
Center and its strong and experienced professional staff.

Our partners are of world-class stature in scalable computing, in
related technologies such as virtual reality and computer graphics,
in all areas of computer science that relate to parallel computing,
in computational mathematics, in forefront applications.

Mr. Chairman, you have asked how the new competition has af-
fected our Center. It has certainly diminished the strong and fruit-
ful collaboration with the others that went by the rubric of
"Metacenter."

It has created a sense of uncertainty about the future in the
minds of some of our professional staff. Naturally the realization
that this is a very complex and serious competition with very high
stakes has generated an intense concentration on winning, distract-
ing us somewhat from our normal mission.

In the long term, I am sure that the intercenter cooperation will
re-establish itself. Nevertheless, we share a commitment to the
idea that a partnership of the kind that we envision is a logical
next step for the Cornell Theory Center. It will transform our pro-
gram into something quite new, something that will serve the best
interests of the community devoted to computational science, and
the broader community of research in basic science.

[The prepared statement of Dr. Kalos follows:]

Testimony of Malvin H. Kalos on the NSF Partnership for Advanced
Computational Infrastructure

before the

Subcommittee on Basic Research of the Committee on Science

March 19, 1996

Mr. Chairman, it is a privilege to be invited to comment on the NSFs new pro-
gram of Partnerships for Advanced Computational Infrastructure.

I am Malvin Kalos, the Director of the Cornell Theory Center, one of the four cur-
rent NSF supercomputer centers, and a Professor of Physics at Cornell University.
I have been a computational scientist for more than forty years, and at present am

105

the Vice Chairman of the Division of Computational Physics of the American Phys-
ical Society.

Our Center was founded with the premise that computation can have a profound
impact upon all the sciences and engineering. From the beginning, the Center took
a leadership role in scalably parallel computing as the ultimate source of the most
processing power and memory. We understood that orders of magnitude increases
in computational power could only come from harnessing large numbers of commod-
ity microprocessors, rather than smaller numbers of specially fabricated computer
chips. Now computational science is undergoing a revolution with the advent of
practical scalable computing. Computing resources have increased by three orders
of magnitude in speecf and memory during the past decade. The 512-processor SP
at Cornell is one of the most powerful computing environments available today. We
have seen leading researchers in our community cross new thresholds of capability
to do qualitatively new science — that is, to attack new problems which could not be
considered before and old problems in ways that provide new insights.

In the end, a program like this and our Center can only be judged by the quantity
of important scientific knowledge and understanding which could not have been ac-
complished without it and by the impact of this new knowledge upon our society.

An international team of scientists collaborated to investigate the rapid activity
of the enzyme acetylchoinesterase. The enzyme breaks down the neurotransmitter
acetylcholine diffused across nerve cell synaptic gaps. Its three-dimensional crystal
structure was solved several years ago, and showed the active site of the enzyme
to be a long, narrow channel — too narrow to deal rapidly with the job of breaking
down acetylcholine. Yet it is known that acetylcholinesterase acts very rapidly.
Using computing resources at all four of the national centers, the researchers dis-
covered that there was a "back door" to the active site of the enzyme that explained
its ability to work so quickly. Since inhibitors of acetylcholinesterase are important
medications for diseases such as myasthenia gravis, glaucoma, and Alzheimer's dis-
ease, this new insight may lead to more effective drugs to fight these diseases.

Using both the Arecibo radiotelescope for observation and the Cornell Theory Cen-
ter's computational facility for intensive analysis and modeling, Alexander
Wolszczan of Penn State University succeeded in identifying the first planets out-
side of our solar system. Wolszczan was recently awarded the prestigious Beatrice
M. Tinsley Prize of the American Astronomical Society for his discovery.

John Dawson and his research group at the University of California at Los Ange-
les use particle-in-cell methods to study turbulence in complex plasmas. The mem-
ory and speed of the Cornell SP permit his group to perform simulations that are
an order of magnitude improvement over previous machines in spatial resolution,
and so allow them to see interesting and important new effects. They have been
able, for the first time, to simulate successfully the parameters of major Tokomaks,
such as the TFTR at Princeton, which is the largest fusion device that exists in the
United States.

A number of cardiovascular and psychiatric diseases are currently treated with
drugs that act on the neurotransmitter serotonin and its receptors. The cellular re-
ceptor for serotonin is a gatekeeper molecule that recognizes and binds the serotonin
and then transmits the signal to the cell by binding to a special class of trans-
ducers — the G-proteins. Researchers at Mount Sinai Medical Center made a break-
through in modeling the serotonin receptor using CTC resources. The breakthrough
came from modeling the structural changes that occur in the serotonin receptor
when it binds to a ligand and to the G-protein, causing it to carry out its function.
This research shows how G-proteins can be switched on by structural changes in
specific regions of the receptor molecule and is expected to lead to the development
of more effective drugs for a vast range of diseases.

Edmund Bertschinger, Massachusetts Institute of Technology, conducts Grand
Challenge research aimed at the understanding of the formation of galaxies in the
universe. The memory and power of computers such as the SP permit him and oth-
ers to study model systems large enough to capture these large-scale dynamical ef-
fects.

George Karniadakis, Brown University, is studying turbulent flow through direct
numerical simulation in order to resolve flow fields at all scales from first principles.
A fundamental goal is to provide insight into a number of open questions in fluid
dynamics, including large-scale instabilities of turbulent wakes and associated drag
and lift variation in flows past bluff bodies; symmetry-breaking bifurcations and
low-dimensionality in flow-structure interactions; and shear stress control and vor-
ticity transport in wall-bounded flows using both passive and active means of con-
trol such as surface modifications and electromagnetic techniques.

Toichiro Kinoshita, Cornell University, studies the anomalous magnetic moment
of light elementary electrons, whose interactions are primarily electromagnetic and

106

which therefore provide very stringent tests of quantum electrodynamics. A power
series expansion of the magnetic moment gives successive terms that are integrals
over more and more dimensions. They are done by a very sophisticated Monte Carlo
but require much more resources at higher orders. Moving to the SP has permitted
Kinoshita to go to third and fourth order.

We believe that we have also made a singularly important contribution to the na-
tional scientific effort by the depth and quality of our partnership with IBM.
Through this partnership, we rekindled IBM's interest in high-end technical com-
puting and thereby helped to bring their superb technology to the service of science
and engineering. As is often the case, the technical computing world has led the way
in bringing about new technologies for the broader commercial arena; we believe
that IBM's success will help assure a future for computational science.

The essence of pure science is to connect our scientific knowledge in a seamless
web of quantitative understanding. This has become harder as we try to probe into
more and more complex phenomena that cannot be analyzed by the mathematical
tools at our disposal. Computational modeling is essential to fill this need.

Many areas of science involve this kind of systematic connection among different
phenomena at different scales of length or energy. All aspects of chemistry, biology
and medicine, the physics of materials, astrophysics and many others require this
approach.

Computational science is now also an essential tool in experimental science. The
most advanced scientific instruments, optical and radio telescopes, particle accelera-
tors, and computers themselves are studied, designed, optimized, and verified with
computer simulation. Data collection is usually automated, and the reduction to
comprehensible data sets may involve enormous computations. Exchange of large
data sets will require very heavy use of future high capacity data networks.

Now the entire scientific and engineering community of the country has the op-
portunity to exploit the new tools of computational science. Students and young sci-
entists, who are always the very heart of any important scientific change, are in-
volved. We share NSFs vision that education of tomorrow's young scientists includes
bringing them into the process of exploiting the nation's most advanced tools, in-
cluding the computational resources of the National Centers. They will carry the
message to the rest of our society and to the future.

Another vital role of computational science is that of permitting quantitative con-
nections among different disciplines, that is, in supporting multidisciplinary re-
search. Every one of the large problems that confront our society, and to whose solu-
tions we expect science to contribute, is in some sense a multidisciplinary problem.
Issues of the environment and medicine, to cite only two, involve many sciences —
chemistry, physics, engineering, fluid flow, biology, and materials.

Bringing the knowledge from these fields together to make quantitative pre-
dictions about the effects of some technological or regulatory proposal would be ut-
terly impossible without the use of high-performance computational modeling, the
indispensable natural language of quantitative multidisciplinary research.

The supercomputing community now finds itself at a major crossroads. To exploit
these new machines, a major re-tooling of software and algorithms will have to take
place.

Science and its application to societal problems involve the national community.
Bringing to bear the transformation made possible by computational science in the
most complete and positive way requires that its techniques and strategies be
learned, used, and shared by the widest possible group of researchers and educators.
All of these are necessary, and the appropriate level and balance among them is es-
sential.

In response to the new NSF program, but also to assure that these advances are
sustained and extended, the Cornell Theory Center is increasing its collaborations
with computer science and computational mathematics to meet the challenge of this
re-tooling. We are therefore creating a National Alliance for Advanced Computa-
tional Infrastructure. This will comprise a new network of collaborating organiza-
tions that will bring their varied talents and resources to the Alliance along with
the world-class supercomputing infrastructure of the Theory Center and its strong
and experienced professional staff. Our partners are of world-class stature in scal-
able computing; in related technologies, such as virtual reality and computer graph-
ics; in all areas of computer science that relate to parallel computing; in computa-
tional mathematics; in forefront applications.

You have asked how the new competition has affected our Center. It has certainly
diminished the strong sense of collaboration with the others that went by the rubric
of "MetaCenter." It has created a sense of uncertainty about the future in the minds
of some of our professional staff. Naturally the realization that this is a very serious

107

competition with very high stakes has generated an intense concentration on win-
ning, distracting us from our normal mission.

Nevertheless, we share a commitment to the idea that a Partnership of the kind
that we envision is a logical next step for the Cornell Theory Center, one that will
serve the best interests of the community devoted to computational science and the
broader community of research in basic science.

Mr. BOEHLERT. Thank you, very much.
Dr. Smarr?

STATEMENT OF DR. LARRY L. SMARR, DIRECTOR, NATIONAL
CENTER FOR SUPERCOMPUTING APPLICATIONS, AND PRO-
FESSOR OF PHYSICS AND ASTRONOMY, UNIVERSITY OF ILLI-
NOIS AT URBANA-CHAMPAIGN, ILLINOIS

Dr. Smarr. Thank you very much.

I am Larry Smarr. I am the Director of the National Center for
Supercomputing AppHcations at the University of Ilhnois at Ur-
bana-Champaign. I am also Professor of Physics and Astronomy.

I would like to make a few points that have been brought out by
the questions today. I will leave my written testimony to the
record.

I think one of the interesting things as I look back over the last
ten years, since I was very involved in getting this program started
in the first place — I was one of those folks who was mentioned ear-
lier that had to go to foreign shores to get access to American-built
supercomputers — is that in those days really it was just the na-
tional laboratories that had these capabilities.

There was a lot of uncertainty among folks like you in similar
Congressional hearings about whether universities would in fact be
able to rise to the occasion, and whether the NSF itself would be
able to handle something that traditionally had been dealt with my
the DOE or by the military.

I think that the fact that, say at our center alone which is en-
tirely run by the University of Illinois, the University staff, grad-
uate students, post-docs, faculty, and the fact that we have had
over 9000 users during that last ten years fi-om 500 different insti-
tutions, the fact that we have corporate partners from all sectors
of the economy speaks to the fact that the National Science Foun-
dation was in fact able to rise to the occasion.

The entire national network, the NSF Net that we take for
granted today and that is all commercialized even, arose completely
from the backbone that was used to tie those centers together at
56 kilobits. That speed is less than what you get into your home
today.

Remember the — I mean, ten years ago we were talking about
this national backbone being, today, of lower speed than ISD into
the home. And in fact our first supercomputer had the speed that
is about equal of what Nintendo is going to bring out next month
in their new Ultra-64. The 8-year-olds are going to have
supercomputers in their basement playing videogames.

That is what this dramatic technology does. There is no other
technology like it. At our center alone we have been able to sustain,
at essentially the same amount of dollars per year spent on hard-
ware, a 75 percent compounded annual growth rate in computer ca-
pacity.

108

We have gone through three complete generations, architectures,
from the original shared Vector machines to the massively parallel,
to the new microprocessor based systems that are going to combine
shared memory and ease of programming with the scalability of
massively parallel machines.

One thing I would like to read into the record is the appendix
from the Hayes Report — and I think all four directors want that to
happen — that give the major accomplishments of the entire NSF
Centers Program both in technology development, outreach, and in
the scientific research enablement of remote users throughout the
university community. There is a lot of work that went into prepa-
ration of that document that I think adequately will answer some
of the questions about the successes of the program.

I would like to move next to this notion that it is all about
supercomputing. That certainly was the case ten years ago. It is
not the case today.

The centers have evolved and matured and their role within the
context of the national program has changed dramatically.

For instance, the NSF center dollars, the Cooperative Agreement
we are talking about, the $65 million available each year from the
Division of Advanced Scientific Computing, at our Center is
matched dollar for dollar by other sources that grew up as a result
of having that money there. That came from the State. It comes
from corporate partners. It comes from vendors. It comes from
other agencies.

So that there has grown up around the core program a whole set
of partners. So the partnering idea that we are seeing in the solici-
tation is nothing new. It has organically grown out of the success
and maturation of the original center concept.

The network is now accelerating the need for that.

Furthermore, from the very beginning it was clear that visualiza-
tion and now Virtual Reality techniques are needed to be able to
understand the massive amount of numbers that are generated
from the supercomputers.

So Virtual Reality development itself has become a major aspect
of ours and other centers, as well as software development.

Many of you are I am sure aware that one software product,
NCSA MOSIAC, that came from our Center has completely trans-
formed the world in a matter of just a couple of years.

Before this, the Worldwide Web was something I do not think
any of us had ever heard about. With the ease of use of this one
software product, millions of users worldwide in probably just 18
months started using this to lay the basis for one of what Wall
Street considers the most successful technology transfer program in
probably the history of federal funding, which is the development
of companies like NetScape's SPYGLAS, the licensing of over 40
companies of this product including MicroSoft, and now this battle
between NetScape and MicroSoft.

All of that came from one set of mainly students working at our
Center.

So the ability to keep America on the cutting edge of this new
world of networks, and software, and the Web, the Centers have
played a decisive role in all of that.

109

The same goes — and I will not go into detail — for the K through
12, the outreach to communities, the work with industrial partners,
and indeed the Federal Government. There are 16 Federal agencies
that are going together to learn how to help in the streamlining of
the Federal Government system using these techniques at NCSA.

So what is wrong with the current picture?

That is what I am hearing.

I think the point is that, with networking you are having a phase
change in the way this country is put together, whether you are in
business, or whether you are in the government, or whether you
are in health care, or whether you are in education, or whether you
happen to be in high-performance computing.

That is, that distance is rapidly disappearing.

So the notion that a Center being in one geographical spot is be-
coming an anachronism. In fact, what is going to happen is in the
way you see all the modem management books for industry emerg-
ing is that of Virtual Teams that come together, go apart, are sup-
ported by the Network technologies, and the fact that we have now
these Lego-like units of microprocessors that can go from the
desktop machine to mid-level machines to the very highest leading
edge machines, that are distributed across and almost like a sea
of processors across this network.

That is the world. It has nothing to do with this solicitation. The
world has changed, and largely because of the success of the first
ten years of the program.

The way I interpret the NSF's solicitation is that they are now
saying. Given this new world, given that we have a fixed amount
of dollars each year in the Division of Advanced Scientific Comput-
ing, how can we create Virtual Centers that take advantage of this
new world and all of the ideas, all of the smart people that are not
in Champaign-Urbana?

How do they become part of the Center?

Right now the fundamental problem is. If I wanted to fund an
effort that was at some other university directly, rather than spend
it in Champaign- Urbana, I really cannot do that within the cur-
rent structure of the program.

The new program would envision that a portion of the coopera-
tive agreement that would come to me as PI at my center would
in fact be spent out in the field at these other partnering institu-
tions and not at the center.

I think it is only when you can actually get resources, dollar re-
sources, out of the partnering centers that the kind of tight cou-
pling that Ed Hayes referred to actually comes into being, because
once you are doing that, you also have the people at those other
centers that are involved in the management of your own central
center.

I think that that shared management of shared brains around
the table from a number of the very best places in the country is
a very different vision than the current one in which there is sim-
ply one cooperative agreement that has been in one place, and then
there are other programs.

So I think that is one of the key advantages.

The other is that there is this tendency to say, well, we are going
from four centers to two centers. I do not see that at all.

110

Technically, yes, to get within the same budget money to put out
in the Partnership Centers you may well have to reduce the num-
ber of leading edge centers, but you are gaining other centers in
the process — namely, the partnering centers.

They are becoming an integral part in a virtual teaming ap-
proach of this new PACI leading edge center. So we could end up
with Centers of Excellence in Virtual Reality, and Mid-Level Per-
formance, and Reaching Out in K Through 12, that are actually
more than we currently have.

The expense to do that within a fixed budget is to reduce the
number of leading edge centers.

All I can say is that the solicitation has generated more new
ideas, more good feeling among the community at large, than any-
thing I have seen in ten years. I think that the kinds of concerns
you have that we do not break the well-functioning current system
are well placed, and v/e have to be very careful that that is taken
care of during the solicitation.

The best protection for that is a strong merit review process. I
think and hope that the committee today will reinforce the appro-
priateness that merit review through the NSF system is the best
way to make sure that there is a smooth transition and that none
of the services to the national community are less, but in fact
greatly increased.

[The prepared statement of Dr. Smarr follows:]

Ill

Congressional Testimony

Prepared for
Chairman Steven H. Schiff

Subcommittee on Basic Research

Committee on Science

U.S. House of Representatives

On the National Science Foundation's

Partnership eor Advanced Computational Science

Solicitation

NCSA

March 19, 1996

Dr. Larry L. Smarr
Director

National Center for

Supercomputing Applications

University of Illinois at Urbana-Champaign

605 East Springfield Ave.

Champaign, Illinois

112

L'ni\'ersit\' of Illinois
at Lrbana -Champaign

National Center for
Supercomputing Applications

Larry L, Smarr
Director

5365 Beckman Institute

for Advanced Science

and Technology

Drawer 25

405 North Mathews Avenue

Urbana, IL 61801

FAX

217 244-8195

Inteniet
lsmarr(Q)ncsa. uiuc.edu

217 244-0078

March 19, 1996

The Honorable Steven H. Schiff
Chairman, Basic Research Subcommittee
U.S. House of Representatives
Committee on Science
Suite 2320 Raybum House Office Building
Washington, DC 20515-6301

Mr. Chairman:

Mr. Chairman, it is an honor to be here this afternoon. I am going to talk with you
today about the National Center for Supercomputing Applications at the University of
Illinois at Urbana-Champaign.

Specifically, I would like to talk with you about how NCSA is participating in the
National Science Foundation's Partnerships for Advanced Computational Infrastructure
solicitation.

In Section 1 of my written testimony I have provided you with an overview of NCSA's
programs. This section contains information on NCSA's history, funding, research
tools, technology and knowledge transfer, our Fortune 500 industrial partners, and the
Federal Consortium.

Section 2 provides a response to two of the questions you mentioned in your letter of
invitation.

Attachment 1 highlights major scientific accomplishments of the NSF Supercomputer
Centers Program.

Again, thank you Chairman Schiff for the opportunity to participate in today's hearing.

Eh-. LafTyL- Smarr

113
Table of Contents

Section 1 — Overview of the National Center for Supercomputing Applications 2

Background 2

Funding Sources 2

New Tools for Computational Science and Engineering 3

Technology Transfer 3

Knowledge Transfer 3

Fortune 500 Industrial Partners 3

NSF/NCSA WWW Federal Consortium 4

Section 2— Impact of the NSF Solicitation on NCSA
Attachment 1 — NSF Centers' Accomplishments

Dr. Lairy L Smarr. NCSA
Congressional Testimony

114

Section 1 — Overview of the National Center for Supercomputing Applications

Background

Under the leadership and guidance of the National Science Foundation the National Center for Supercomputing
Applications (NCSA) at the University of Illinois at Urbana-Champaign has evolved into a scientific research center
built around a national services facility. All aspects of NCSA are carried out within a major research university. At the
founding of the NSF Centers program 10 years ago, there were those who questioned whether a university could
successfully manage such a critical, national-scale program. History has shown that the integration of the program into
the university environment has resulted in much of the uniqueness and synergy leading to rapid advances in scientific
knowledge, application software development and technology transfer.

In its leadership role, NCSA is developing and implementing a national strategy to create, use, and transfer advanced
computing and communication tools and information technologies. These advances serve the center's diverse set of
constituencies in the areas of science, engineering, education, and industry.

When the center opened to the research community in 1986, researchers using the supercomputers were from the
traditional disciplines of physics, chemistry, materials, and astrophysics. But researchers in other disciplines quickly
realized they could also transform their fields by using supercomputing. Today's growth areas include biology, medicine,
environment, computer science, commercial and fmancial databases, and networked World Wide Web multimedia asset
management

Academic and industrial users from across the nation have utilized NCSA's leading edge supercomputers to enable
advances in their research. As a result of rapid changes in computing technology, NCSA has been able to continually
increase supercomputer capacity and capability while keeping dollar expenditures relatively constant. Specifically,
NCSA capacity of supercomputing time made available to national users has increased at 75 percent per year, resulting
in a 1,000 fold increase in capacity between the center's opening in 1986 and the end of the current cooperative
agreement in FY97. To utilize this rapid growth in computing power, NCSA has worked with users to develop and
migrate application codes through three distinct phases of supercomputer architectures: shared memory vector
processors; massively parallel processors; and scalable shared memory RISC microprocessors. During this period, nearly
9,000 users from more than 500 academic institutions and corporate laboratories in all 50 states have made use of
NCSA's computational resources. In addition, hundreds of university level courses in computational science and
engineering have used NCSA supercomputers. See Attachment 1 (Major Accomphshments of the NSF Supercomputer
Centers Program), detailing scientific highhghis and user accomplishments. The attachment is being jointly submitted by
all four NSF centers.

NCSA is dedicated to improving the competitive position of American industry by partnering with leading corporations
in a variety of sectors in the US economy to partner with NCSA. These Fortune 500 companies have selected NCSA lo
learn about all aspects of cutting-edge hardware and software, virtual prototyping, visualization, networking, and
databases.

NCSA works closely with computer vendors and national leaders of academic computer science to bring users the most
advanced methods in high-performance scalable computing. The center currently maintains these supercomputers:
Hewlett-Packard/CONVEX Exemplar, Silicon Graphics POWERCHALLENGEarray, and Thinking Machines CM-5.
NCSA has allowed its vendor partners to gain access to national leaders in science and engineering and, as a result, the
vendors have produced machines that are more useful to the American research community.

Funding Sources

NCSA relies on its cooperative agreement from NSF as the foundation of the center's efforts. However, NCSA has
aggressively leveraged the core NSF funding providing a dollar to dollar match from other sources. This has resulted in
an additional $132.8 million over 10 years being added to NCSA's program beyond the NSF core funding. Sources of
these leveraged funds are other federal agencies, corporate partners, the University of Illinois, and the State of Illinois.
This additional funding has been critical in expanding NCSA's offerings beyond a central supercomputing facility and
for developing world recognized staff in software development, scientific applications, and virtual environments.

Dr. Lany L. Smart, NCSA
Congressional Teslimony

115

New Tools for Computational Science and Engineering

To support the large number of remote users NCSA has developed a broadly based scientific research center of
application researchers housed in the Beckman Institute at the University of Illinois at Urbana-Champaign. These staff
members act as a bridge to their disciplines and have been a constant source of innovation in the use of new technology
10 empower new applications. NCSA has application scientists in the fields of astrophysics, biomedical science,
chemistry, chemical engineering, computational biology, computational fluid dynamics, condensed matter physics, earth
sciences, gravitational sciences, humanities/fine arts, information systems, mathematics, computer science, radio
astronomical imaging, and structural engineering. For example, one of NCSA's application scientists is PI on the joint
agency (NSF, ARPA, NASA) Digital Library Initiative at the University of Illinois at Urbana-Champaign.

NCSA's Virtual Environments Laboratory is one of the world's most advanced virtual reality (VR) research laboratories
available to academic and industrial researchers. The lab consists primarily of three projection-based modes of VR — the
CAVE™ and the ImmersaDesk™ (both developed by the Electronic Visuahzation Laboratory [EVL] at the University of
Illinois at Chicago) and the NlIAVall (developed by EVL in collaboration with NCSA and the Laboratory for
Computational Science and Engineering at the University of Minnesota).

Technology Transfer

In addition to developing new software tools and making them freely available to the academic community, NCSA has
had a significant impact on commercial software development and products. NCSA is proud of its proven record of
catalyzing new products and companies. NCSA software developers who joined InterCon Systems Corp. in 1987 started
a long-term product line based on NCSA Telnet™ . Similarly, former NCSA developers founded Spyglass Inc. in 1990
with software based on NCSA Image™. The 1992 release of the World Wide Web network browser NCSA Mosaic
created millions of Web users and both laid the technology groundwork and created a market for subsequent
commercialization. This commercialization was accomplished via two routes. First, NCSA software developers joined
companies such as Microsoft, Spry, Spyglass, and Netscape. Second, UlUC/NCSA intellectual property is licensed
through Spyglass to 40 companies, and incorporated into more than 100 products. The economic results are represented
by the example of just two of these companies. Spyglass and Netscape, which in the past twelve months have attained a
market valuation as high as $5 billion.

Knowledge Transfer

NCSA understands that transferring necessary technical skills and information is vital for prolonged growth and success
of any research activity. This transfer is accomplished by providing computational science, networking technologies, and
information technology training to fellow researchers, students, other educators, lifelong learners, government and
community organizations, and members of the business community.

NCSA addresses science and curriculum issues through programs designed not only to give a technical jump-start to a
school but also to provide continuing support to educators and their administrators. Current major programs include: the
Education Affiliates (EA) program, the Resource for Science Education (RSE) program, the Networking Infrastructure
for Education (NIE) project, EduLinK-12 (IlUnois Education Link), Tech Corps Illinois, and Illinois Learning Mosaic
(ILM, a World Wide Web-based educational information resource).

NCSA's community networking and technology transfer program works with various communities to deploy computing
and communications technologies and applications at the local, state, regional, and national levels. The center's CCNet
(Champaign County Network) collaboration with its host community has received national recognition as an innovative
application of the National Information Infrastructure and as a model for other communities. Chicago Mosaic, a
collaborative project with the City of Chicago, is investigating the use of World Wide Web technologies lo communicate
among city government, the citizens of that community, and the rest of the world.

Dr. Lany L. Sman, NCSA
Congressional Testimony

116

Fortune 500 Industrial Partners

Current NCSA industrial partners include Allstate, AT&T, AMR, Caterpillar, Dow Chemical, Eastman Kodak Co., Eli
Lilly & Co., FMC, J. P. Morgan, Motorola Inc., Phillips Petroleum Co., Schlumberger, Sears, Roebuck and Co., Tribune
Co., and United Technologies. Since 1986 sixteen partners have added 540,672,913 to NCSA's programs in order to
learn about the evolving high performance computing technologies, to apply them to solving breakthrough industrial
applications, and to support technology research at NCSA. Their investment in this program has enabled each company
to enhance technical and management skills and to explore the leading edge technologies which are needed for them to
help stimulate this country's economic viability. As a result of their successes in this program, NCSA and the University
of Illinois at Urbana-Champaign have gained a reputation for excellence among our nation's industrial leadership.

NSF/NCSA WWW Federal Consortium

A group of 16 US Federal agencies, assembled through the efforts of the National Science Foundation, came to NCSA to
learn how to apply new technologies in their efforts to stfeamline internal, cross agency, and multi-level government
processes and communication. Member agencies include the following: Nuclear Regulatory Commission; National
Library of Medicine; Cenual Intelligence Agency; Intelligence Services (COSPO); National Science Foundation;
National Institute of Standards and Technology; Department of Education; Defense Technical Information Center;
National Oceanic and Atmospheric AdminisU'ation; National Security Agency; National Cancer Institute; National
Institutes of Health; National Aeronautics and Space Administration; US Geological Survey; National Biological Service
and Bureau of the Census.

Dr. Larry U Smarr. NCSA
Congressional Testimony

117
Section 2— Impact of the NSF Solicitation on NCSA

Q: "What impact is the new solicilation having on the current centers?"

A: NCSA has partnered with a number of groups during the past 10 years including US industry and other R&D
facilities. In response to the solicitation NCSA is expanding the scope of its partnerships and subsequent projects.
Partnering with other institutions is a key component of the NSF solicitation. Therefore, NCSA is aggressively pursuing
exciting new working relationships with computational scientists, computer scientists, academic institutions, R&D
laboratories and regional centers to prototype the US computational and information infrastructure of the 21st Century.

Also, NCSA is developing new and innovative ways to manage a larger number of partners and highly complex projects.
The end result will be a stronger, national-scale laboratory for the creation and management of "virtual teams," which are
becoming essential to the emerging knowledge worker communities in American industry, government, universities,
health care, and education.

The solicitation has not led to a negative impact on the current daily operations at NCSA. The PACI mission has
heightened vendor interest in increasing their commitment and partnerships with NCSA. In addition, the solicitation has
led to a closer working relationship with NCSA's end users.

In conclusion, the solicitation has challenged NCSA to raise its sights to see what the center can further accomplish to
accelerate national objectives.

Q: "How are the current centers working to participate in the new solicitation proposals?"

A: As requested by NSF, NCSA's management is writing a preproposal based on the PACI mission statement and NSF
guidelines. NCSA is competing to become one of NSF's leading edge sites in the PACI program.

NCSA has taken the following steps to seek out partnerships:

• Discussed NCSA's new vision with potential parmers (including DOE laboratories, universities, consortia, research
groups, other NSF research centers, and state and regional centers)

• Solicited innovative ideas from potential partners

• Matched NCSA's vision with potential partners and their ideas

• Aligned all goals and objectives with PACI mission statement

NCSA has found the challenge of the PACI mission exciting. The center has also witnessed many innovative ideas
emerge from the broader community stimulated by the solicitation. As long as this process remains under the firm
control of NSF's merit review system, NCSA believes the result will be a stronger, more broadly based national
program.

In conclusion, NCSA believes that the PACI program is critical to the future of America's research system. The PACI
program will allow for critical early experience with the advanced computational infrastrucnire that will support 21st
Century scientific and engineering research.

Dr. Lany L Smair, NCSA
Congressional Testimony

118

Attachment 1
Major Accomplishments of the NSF Supercomputer Centers Program

From the Report of the Task Force on the

Future of the NSF Supercomputer Centers Program

September 15, 1995

High Performance Computing Infrastructure and Accomplishments
Introduction

Important Technology Accomplishments

HIGH PERFORMANCE COMPUTING
Supercomputer Usage ai NSF Centers
Architectures and Vendors
Center Program Chronology
National Access to Vector Multiprocessors
Achieving Production Parallelism
Early Migration to the UNIX Operating System
Early Access to Massively Parallel Computers
Superlinear speedup on heterogeneous processors
Workstation Clusters

PORTABLE PARALLEL PROGRAMMING TOOLS
Prototype Parallel Programming Enviroiunents
Extensions of PVM
Scalable Libraries

STORAGE TECHNOLOGIES

AFS-Establishing a National File System
HDF-Creating a Standard File Format
Migrating to a Standard Archiving Software
Development of high-density magnetic media

NETWORKING

Evolution of NSFNET

High Performance LANs

Gigabit Testbeds

Secure Networks

New Science Enabled by Networks— Telemicroscopy

Nil Testbeds

VISUAU7ATION AND VIRTUAL REAUTY
Development of Scientific Visualization
Virtual Reahty Impacts Industrial Design
Development of Inunersive Science Projects
Virtual Reality over ATM networks
Alpha Shapes, Biomolecules, and Cosmology

DIGITAL LIBRARIES AND INFOSERVERS
Digital Libraries
Scalable Information Servers
The Rise of the MosaicAVWW Information Infrastructure

DESKTOP SOFTWARE
Connectivity Tools
Collaboration Tools

2
2
3
4
4
4
5
5
6

6
7
7
7

7
7
8
9
9

10
10
11
11

12
12
13

14
14
16
16
16
17

17
17
18
18

18
19
19

119

Graphics Tools 19

Scientist's Workbench 20

Accomplishments in Education and Outreach 2 1

EDUCATION 21

Researchers and S tudents 2 1

Supercomputer Centers Educational Activity Support Sununary 21

Outreach to Educators 23

OUTREACH 24

Apphcation of Scientific Computation and VisualiTation to Industrial Production 24

Impact on Vendors of High Performance Computing Equipment 25

Stimulation of New, Computationally Dependent Ventures 26

Development of Nationally Valuable Reservoirs of Skill 27

Community Service 27

Important Science and Engineering Accomplishments 2 8

Summaries of computationally interesting problems in the NSF Centers Program by the

national science and engineering communities: 2 8

QUANTUM PHYSICS AND MATERIALS 28

Phase Transition in QCD 28

Phase Transitions of Solid Hydrogen 28

Prediction of new Nanomalerials 29

Theory of High Temperature Superconductors 29

Magnetic Materials 30

Understanding Glass 30

BIOLOGY AND MEDICINE 30

Crystallography 31

Folding Proteins using Artificial Intelligence 31

Protein Kinase solution 31

Molecular Neuroscience-Serotonin 32

Molecular Neuroscience-Acetylchohnesterase 32

Kinking DNA 33

Antibody-Antigen Docking 33

Tuning Biomolecules to Fight Asthma 34

Virtual Spider and Artificial Silk 34

Heart Modeling 34

ENGINEERING J5

Ultra-high-strengih Steels 35

Continuous Casting of Steel 35

Beverage can design 35

Designing a Leakproof Diaper 36

Bone Transplant Bioengineering 36

Improving Performance with Ribleis 36

Designing Better Aircraft 37

Crash Testing Street Signs 37

EARTH SCIENCES AND THE ENVIRONMENT 37

Detoxification of Ground Water 38

Sage Grouse-Endangered Species and the US Army 38

Storm modeling/forecastmg 38

Los Angeles Smog 39

Upper Ocean Mixing 39

Simulating Climate using Distributed Supercomputers 40

120

PLANETARY SCIENCES. ASTRONOMY, AND COSMOLOGY 40

Comet Collision with Jupiter 40

Discovery of First Extrasolar System Planet 41

Building the BIM A Radio Telescope 4 1

Pulsar Searching and Discovery 42

Accretion Disks Around Black Holes 42

Black Hole Colhsion Dynamics 42

Largest cosmological simulation 43

EVOLUTION OF THE METACENTER CONCEPT 43

Recognition Accorded NSF supercomputer Users and Projects 44

Acronyms 46

121

High Performance Computing Infrastructure and Accomplishments

Introduction

The NSF Centers Program was established to provide access to high performance
computers (supercomputers) for the broad Science and Engineering Research Com-
munity. The program has evolved from one comprising independent, competitive,
and similar computer centers to one including more cooperative and diverse activi-
ties. Coordinating the mission of the individual centers has increased the diversity
of computer architectures available to the research community, and has accelerated
outreach to segments of the community which had not before been able to use the
power of high performance computers. At the same time, competition between cen-
ters has been managed by NSF and its advisory committees to the advantage of the
engineering and science communities which the program was established to serve.

Building on each center's tradition of providing a stable source of computer cycles
for a large community of scientists and engineers, the centers have evolved into a
unique resource to test which diverse computer architectures best match the most
demanding problems posed by the community of university researchers and to de-
velop the necessary supporting software and algorithms. For example, this approach
has enabled the centers to test which applications can be efficiently served on newly
developed systems using clusters of the new generation of workstations that are
now being introduced. Such experiments are enabled by the open environment
characteristic of the program.

During the first decade of the centers program, major improvements in the delivery
of high performance computing have been developed, mairUy by American com-
puter researchers and companies. But advances in computing technology have been
matched in equal measure by improvements in computer networking, and as a con-
sequence the NSF Centers have been a primary focus for accelerating the evolution
of the Internet via NSFnet, NREN, and the still evolving broad-band width tech-
nology.

In this appendix, we provide detailed examples of the centers activities.

-1

122

High Performance Computing Infrastructure and Accomplishments

Important Technology Accomplishments
High performance computing

Originally set up in 1985 to provide national access to traditional supercomputers,
the NSF Centers have evolved to a much larger mission. The Centers now offer a
wide variety of high performance architectures from a large array of American ven-
dors. No longer just adopting technology from the national labs, the NSF Centers
Program has become a pioneering vanguard of technology - a model for other agen-
cies with a vested interest in the high performance computing to emulate. This to-
day is dominated by research efforts in software, with vital collaborations with com-
puter scientists, focusing on operating systems, compilers, network control, mathe-
matical libraries, and programming languages and environments. The feedback to
the leading US vendors is increasing the usefulness of their product offerings to the
scientific and engineering communities, while making them more competitive.

Supercomputer Usage at NSF Centers

Fiscal Year

Active Users

Usage in CPU
Hours

1986

1336

29,485

1987

3,299

95,751

1988

5,042

121,615

1989

5,967

165,960

1990

7,357

250,628

1991

7,723

361,073

1992

8,252

398,931

1993

7,735

910,088

(Usage is in normalized CPU hours, based on comparative performcmce tests. The astounding leap in ca-
pacity in 1993 is mainly a result of the introduction of new computing architectures to solve the most
demanding of computational problems — the Grand Challenges. The slight decrease in the number of
users is the result of a concerted effort by the Centers to assist many of their users with small memory or
CPU-time requirements to meet their computational needs by the increasingly powerful workstations of
the mid-90's. The greatest benefactors of the increase in massively parallel cycles are the scientists and
engineers addressing the problems with the greatest computing demands)

Architectures and Vendors
The national community has been offered access to a wide and frequently updated set
of high performance architectures since the beginning of the NSF Supercomputer
Centers Program. The current rate of change of the types of architectures, and the
number of vendors offering them, is probably near an all time high. We are in a pe-
riod of ferment which the science and engineering communities sort out the choices
for finding an architecture that matches their various computational problems. A list
of architectures that the NSF Centers Program has offered would include: single and
clustered high performance workstations or workstation multiprocessors, minicom-
puters, graphics supercomputers, mainframes with or without attached processors or
vector units, vector supercomputers, and SEvID and MIMD massively parallel proces-

•2-

123

High Performance Computing Infrastructure and Accomplishments

sors. Similarly, the list of current vendors whose top machines have been made
available would include IBM, DEC, Hewlett-Packard, Silicon Graphics, Sun Microsys-
tems, Cray Research, Convex Computer, Intel Supercomputer, Kendall Square,
Thinking Machines, nCUBE, Alliant, Floating Point Systems, ETA, Stellar, Ardent,
and Stardent.

Center Program Chronology

Milestone or Event

Description

1986

5 NSF Supercomputer Centers become
operational

Cornell 1 heory Center

National Center for Supercomputing Applications

Pittsburgh Supercomputing Center

San Diego Supercomputer Center

John von Neumann Center

1988-1989

Renewal Review

1989

Renew 4 NSF Supercomputer Centers
(5yrs.)

CTC, NCSA, l^C, SDSC

1990-1992

ASC Advisory Committee report
completed

strong recommendations for adding parallel systems to
accompany the stable, production vector systems

1991

8 Vector Supercomputers operating
3 Scalable parallel systems operating

IBM 3090 6 processors (2)
Cray YMP 8 processors (2)
Alliant FX80 8 processors
Cray 2S 4 processors
Cray YMP 4 processors
Convex C240 4 processors
Intel iPSC/860 32 processors^
TMC, CM2 32,000 processors (2)'''

1992

7 Vector Supercomputers operating
9 Scalable parallel systems operating

Joint Planning Initiated

Cray systems remain

IBM ES9000/900 (Upgrade 3090)

Alliant 2800 (upgracie FX80)

Convex C3880 8 processors (Upgrade C240)
Intel iPSC/860 upgrade 64 processors)t
nCUBE2 128 processors!
TMC, CMS 512 processorsT
KSRl 64 processors!
DEC Workstation Ouster (2)t
IBM Workstation Cluster
IBM PVS 32 processors

Initial meeting at SDSC Fall 92

1993

Joint Activities Began

6Vector Supercomputers operating
13 Scalable parallel systems operating

Meeting at PSC
First joint proposals to PPRP
First of Joint projects
Cray C90 16 processors
(other Crays, IBM, Convex stay)
Intel Paragon 400 processors upgrade t
KSRl upgrade 160 processors!
Hewlett Packard Cluster^
MasPar 2 16,000 processors!
IBM SPl 64 processors!
Cray T3D 512 processors!

1994

Joint Activities

4 Vector Supercomputers operating
14 Scalable parallel systems operating

Meeting at CTC
Metacenter Regional Alliances - Mar 94
Expansion of joint projects
One YMP changed to C-90, others the same
IBM SP2 upgrade 512 processors upgrade SPl!
Cray T3D 512 processors !
Convex Exemplar 8 Nodes!
SGI Challenge 32 Nodes

^ Majority of funding provided by other Federal agency (ARPA, NIH) or state,
t Donated in full or in part by the manufacturer for extended evaluation

124

High Performance Computing Infrastructure and Accomplishments

The Centers Program provides a stable supply of vector computing cycles needed by
the research community while investing in significant capacity of scalable parallel
systems, capable of ultimately growing to a size necessary for full scale grand chal-
lenge problems. While the numbers of Vector Supercomputers has decreased, the
computing power represented by that group in fact increased substantially. How-
ever, the increase in the scalable parallel systems was much larger, reflecting the
growth potential of this type of computing platform and a strong NSF/ARPA part-
nership.

National Access to Vector Multiprocessors

The NSF Supercomputer Centers established in the mid-1980s brought access to state-
of-the-art supercomputers for the first time in at least 15 years. Indeed, in the 1960s,
only a few universities had such access. This opening of universal access led to an
unprecedented increase in the number of researchers and universities involved in
advancing the frontiers of scientific and engineering research by using high perfor-
mance computing. By the early 1990s, some 15,000 researchers in over 200 universi-
ties had used one of the Cray Research vector multiprocessors or the IBM vector
mainframe in one of the NSF centers. This wide pool of computational researchers
made it possible for the center's program to begin to respond to the demand for paral-
lelism that had been developed in the Computer Science Community, and adopted
by the most adventurous user. The 90's saw the NSF center's program substantially
widen its range of architectural offerings.

Achieving Production Parallelism

The Cornell Theory Center (CTC) became the first member Center of the NSF Meta-
Center to achieve production parallelism on a vector supercomputer, with over 1/3
of its vector supercomputer cycles used for parallelism in 1989. CTC integrated its two
(ES/3090 600) vector supercomputers using a special 200 mbyte/sec hardware inter-
face, allowing parallel jobs the potential of executing across 12 vector processors.
Users used a shared-memory parallel FORTRAN developed by IBM in a joint project
with the Theory Center.

Early Migration to the UNIX Operating System

During the early and mid-1980's the UNIX operating system was widely viewed as
inappropriate for supercomputers for reasons of performance, system management
tools, application development and measurement mechanisms, and security. Cray
supercomputers were run mostly with operating systems that were designed at na-
tional laboratories (LLNL and LANL in particular) and this required extensive local
software support. In 1987, NCSA became the first major supercomputer center to mi-
grate its Cray supercomputer from CTSS (Cray's proprietary time-sharing system) to
UNICOS, a UNIX-based operating system developed at Cray Research for its super-
computers. This move to UNIX was the beginning of a merger between computa-
tional science and computer science, because most computer science research in-
volved the UNIX operating system at that time. Coincidentally, CTC was the first site
to run IBM's high performance UNIX system on its ES/3090 and ES/9000 main-
frames in production.

-4-

125

High Performance Computing Infrastructure and Accomplishments

Early Access to Massively Parallel Computers

Beginning in 1985, CTC provided experimental scalable parallel machines, first the
FPS T-series and an iPSC/1 parallel system, to its user community. In 1988, with the
installation of its iPSC/2 with 32 processors, CTC made early scalable computing
available for production use by the national community. Massively parallel comput-
ing was introduced to the research community beginning with NCSA's CM-2 in
1989. Each Center provided early access to new generation MPPs. The CTC was the
first site to install an IBM SPl and SP2, PSC installed the first Cray T3D, and early
CMS (NCSA), Paragon(SDSC), Ncube(SDSC), and Kendall Square (CTC) machines
were installed. Early access to these machines, enabled by support from ARPA and
NIH, allowed pioneering users to explore the benefits of fine-grained parallelism.
Each Center worked with the user community and the vendors to develop
application codes, which could then be ported to other platforms. In 1992, the CM-5
was added to the program at NCSA as the largest distributed memory parallel super-
computer available to the national academic and industrial communities. From
1992 to the present NCSA has worked closely with national users and the computer
science community to create a wide range of 512-way parallel application codes that
can in 1995 be moved to other large MPP architectures such as the T3D at PSC, the
Intel Paragon at SDSC, or the IBM SP-2 at CTC.

Superlinear speedup on heterogeneous processors

In 1991, PSC was the first site to distribute code between a massively parallel machine
(TMC-CM2) and a vector supercomputer (Cray YMP), linked by a high speed channel
(HiPPI). Experiments on applications as diverse as molecular dynamics, medical
imaging, chemical flowsheeting and gene sequence alignment showed superlinear
speedup (the applications on the linked system ran more than twice as fast on each
system separately). This formed part of the motivation for heterogeneous comput-
ing, as later embodied in the tightly coupled Cray T3D/C90 systems. PSC's T3D was
the first shipped anywhere. PSC developed a set of codes for transferring data be-
tween the Cray and CM2 which later enabled them to coitununicate between the T3D
and C90 at speeds superior to what was available from the vendor.

A similar superlinear speedup was obtained on the CASA gigabit testbed set-up in-
cluded parts of two supercomputers (64 nodes of the 528-node Intel Delta system at
Caltech and one processor of the CRAY C-90 at SDSC, 150 miles away); two HiPPI-
SONET gateways; and a SONET wide-area link between San Diego and Pasadena,
which is itself a prototype undergoing tests in a collaboration between MCI and Pa-
cific Bell. Using an environment for distributed parallel computing called Express (a
product of ParaSoft Corporation), Aron Kuppermarm and Mark Wu (Dept. of
Chemistry, CalTech) did a test calculation of the reaction of atomic hydrogen with
molecular heavy hydrogen (deuterium) at a total energy of 2.5 eV. This problem had
taken 100 hours to solve on the SDSC CRAY Y-MP a year earlier. On the new C-90, it
took 17 hours. On the Delta alone, it took 16. But when the problem was distributed
between the C-90 and the Delta, the whole problem was solved by the two machines
in just under five hours, a factor of 3.3 faster than it could be done on either ma-
chine alone.

126

High Performance Computing Infrastructure and Accomplishments
Workstation Clusters

Given historical trends showing much more rapid improvement in microprocessor
technology than in vector technology, many Centers began exploring workstation
clusters. In 1990, NCSA examined the usage of the Cray Y-MP and determined that a
significant amount of capacity could be gained by moving appropriate applications to
scalar RISC processors. Based on this study and on predictions that microprocessor
technology would surpass vector technology by the mid-1990's, NCSA set up an IBM
RS/6000 cluster as a farm of processors used as stand-alone compute servers. Begin-
ning in 1991, CTC did pioneering work with IBM on clustered RS/6000 workstations
with high-speed, proprietary communications links, including an experimental opti-
cal switch. The information gained during this joint project was used to guide the de-
sign of the SP systems. Also in 1991, PSC set up a cluster of DEC workstations, and
working with Florida State University, significantly enhanced queuing, accounting
and control software and also integrated the AFS file system into this environment.
These clusters all served as computing resources and also as platforms for the devel-
opment of distributed memory message passing codes. These projects generated high
interest in industry, and NCSA trained several dozen industrial sites on the integra-
tion, operation, and management of clusters of microprocessors. NCSA is now mov-
ing to establish dusters of shared memory workstation multiprocessors from Con-
vex/HP and SGI, while PSC will develop applications for Cray's new offering, the J90.
The CTC was given supplemental funding by NSF to build its cluster to 32 proces-
sors, providing a compatible path to its (later) IBM SPl and SP2 systems. The system,
including experimental high-speed switch, was used for production work by the
commuruty, which has now migrated to the SPl and SP2 environments.PORTABLE
Parallel programming Tools

Although the architectures of massively parallel systems differ greatly, the major
time and money investment of the research community (as contrasted to the cen-
ter's personnel) is in developing and converting codes (porting) to operate in differ-
ent environments. The dose cooperation between the NSF Centers via the Meta-
Center and informal contacts among its research users and cooperating agencies
such as ARPA and the various National Labs have resulted in substantial progress
ensuring that labor intensive programming operations need not be duplicated need-
lessly.

Prototype Parallel Programming Environments

Working with the Parascope Group at the Center for Research in Parallel Comput-
ing, an NSF Sdence and Technology Center, CTC developed extensions supporting
new parallel programming paradigms and extensions making ports from one type of
parallel programming platform to another easier. ParaScope is a prototype parallel
programming environment. Both the BIMA and Cosmology GCs at NCSA are work-
ing closely with Indiana University Computer Scientist Dennis Gannon to move ap-
plication codes previously written in FORTRAN to the portable pC++ which is the
model for HPC++, the equivalent of HPF in the C++ world.

127

High Performance Computing Infrastructure and Accomplishments
Extensions of PVM

In the Dome project, Adam Beguelin, one of the original developers of PVM now
working jointly at PSC and in Computer Science at Carnegie Mellon University, is
extending PVM to improve load balancing and fault tolerance. His work is guided by
the experience of PSC's cluster users.

Scalable Libraries

The goal of the ARPA funded Scalable Parallel Libraries project is to develop math-
ematical software libraries for massively parallel processors that are roughly compa-
rable in scope to the math libraries typically available on conventional supercomput-
ers. Michael Heath and his group at NCSA are one team in this multi-institutional
project and they have been developing parallel direct methods for solving sparse lin-
ear systems. For this purpose, they have developed a fully parallel sparse solver for
distributed memory parallel computers. Unlike most other efforts, which have fo-
cused only on factorization, this solver performs all phases of the computation in
parallel mode, including the symbolic preprocessing necessary to reorder the sparse
matrix and distribute it across processors to maintain data locality. With funding
from IBM, CTC staff developed scalable versions of key numerical library routines
for its IBM cluster system; these routines were included by IBM in its ESSL library
product.

Storage Technologies

With the vast increase in both simulation and observational data, the MetaCenter
has worked a great deal on problems of storage technologies. Here again, many of
the biggest areas of progress are in software. The creation of a universal file format
standard, a national file system with a single name space, and a multivendor archiv-
ing software are some of the results of MetaCenter innovation, collaboration with
computer scientists, and with other leading national laboratories. There are even
examples of the Program's computational facilities being used to improve the basic
storage capacity of the physical medium of storage itself.

AFS-Establishing a National File System

PSC recognized that the Andrew File System (AFS), developed at Carnegie Mellon
University with IBM support for a workstation environment, was particularly-well
suited for use in high performance computing, because of its superior security, scal-
ing properties, and manageability. PSC undertook a major program of adapting AFS
to the high performance computing environment which has led to a MetaCenter
wide effort to develop a shared national file system. PSC's AFS enhancements to
Cray's UNICOS are installed at a number of advanced computing centers (SDSC,
NERSC, rPP, LRZ (Germany), ETH (Switzerland) and the University of Stuttgart).
PSC has also extended AFS to multi-resident AFS which enables AFS to be a com-
ponent of a hierarchical storage system, permitting transparent and cost-effective
storage of large amounts of data. These enhancement are installed at other Meta-
Center sites, NERSC, Cray Research, Transarc Corporation, Max Planck Institute for
Plasma Physics, and the University of Cologne. With AFS, distributed applications
across the MetaCenter are now possible. For instance, Paul Dawson (Dept. of Mech.

■7-

128

High Performance Computing Infrastructure and Accomplishments

and Aerospace Engineering, Cornell Univ.), simulating deformations of aluminum,
uses AFS to build a distributed application with modeling performed at PSC and vi-
sualization at the CTC. The SDSC is running an AFS cell supporting the Computa-
tional Center for Macromolecular Structure (CCMS), an SDSC, UCSD, and Scripps
Research Institute collaboration. The charter of the CCMS is the development and
distribution of portable, innovative software for the study of macromolecular struc-
ture. AFS simplifies the cross-institution distribution and maintenance of software
and textual information.

CTC was the first Center to put AFS in production on its HPC systems; in fact, all
CTC production systems, except KSR, and servers are integrated through AFS, in-
cluding its new mass storage environment. CTC has lead in a number of areas of file
system integration, including a project with the University of Michigan integrating
its AFS mainframe port, IFS, into its HPC environment, and joint efforts with
TRANSARC in improving AFS performance over high speed networks, including
an FDDI testbed. New efforts involving TRANSARC include optimization over
ATM and within IBM's SP2 scalable switch.

HDF-Creating a Standard File Format

The NCSA Hierarchical Data Format (HDF), created by Michael Folk and his group at
NCSA has become one of the leading self-describing file formats in the world today.
Many scientific institutions, organizations and programs have adopted HDF as a
standard file format for data exchange and/or archiving. In 1992, NASA selected HDF
as the basis from which to develop an EOSDIS standard data format (SDF). The goal
of SDF is to provide a single, self-describing format for distributing data derived from
approximately 1.2 terabytes of data daily that EOSDIS will eventually produce. Other
examples of HDF adoption include the Institute of Applied GeoScience (seismic
data). Pacific Northwest Laboratory (cancer research). Children's Hospital in Boston
(x-ray images), the UCLA Scientific Visualization Lab. By working closely with many
different user communities to support the harmonization of data models and meta-
data conventions across as many disciplines as possible, NCSA is helping to create a
software foundation for the Nil enabling it to reach its potential to support the broad-
est constituency possible.

Migrating to a Standard Archiving Software

In 1985, the NSF Centers' goal was simply to establish national access for the aca-
demic community to the type of advanced supercomputing and archiving systems
found in the Dept. of Energy national laboratories. NCSA and SDSC duplicated the
Los Alamos computing environment in 1985, including the Common File System
(CFS) archive, while PSC adopted Westinghouse's PDM software that had additional
data migration facilities. As time went on, the Centers began to develop innovation
in storage software. It is the strategy of the MetaCenter to explore alternative techni-
cal approaches to storage at the bits level, while maintaining interoperability
through standard protocols. SDSC, through the DISCOS (Distributed Computer So-
lutions) division of General Atomics (which is now owned by OpenVision), pio-
neered productization of distributed, hierarchical file and storage management ap-
plications software for networked, multi-vendor and open systems environments,

129

High Performance Computing Infrastructure and Accomplishments

based on the IEEE storage model. Working with the UIUC Computer Science De-
partment, NCSA was able to encode data migration and caching strategies into CFS
to improve its ability to minimize disk cache misses. In 1992, NCSA developed CFS-
to-UniTree data formatting and migration tools as well as a suite of archive man-
agement tools. PSC integrated Cray's proven commercial archiving technology
(DMF) with more usable front-end software and with its multi-resident APS. SDSC
collaborates with OpenVision and the National Storage Laboratory in the further
development and stabilization of UniTree as a robust production archival storage
system. SDSC has developed transaction journaUng software that is critically impor-
tant for guaranteeing integrity of the file nameserver. CTC and SDSC are installing
the first generation of a high-performance variant of UNITREE which has been de-
veloped by the National Storage Laboratory, to provide enhanced I/O capability to
balance the increase in installed computing capacity. CTC will continue to work
closely with IBM in the testing and deployment of the next generation of mass stor-
age systems, the High Performance Storage System (HPSS). This will include the ca-
pability of utilizing parallel I/O to speed data transfer to the SP2.

Development of high-density magnetic media

The Center for Magnetic Recording Research (CMRR), located on the campus of
UCSD, has funding from NSF and 21 corporations having enterprises connected
with the storage and retrieval of magnetically written information. The chief tech-
nical problem CMRR has been addressing has been magnetic noise in the metallic
thin-film media used to coat high-density disks. Neal Bertram (Dept. of Electrical
and Computer Engineering, UCSD) is a researcher at CMRR who tackles the prob-
lem of magnetic noise computationally. Magnetic thin films are polycrystalline,
rather than continuous or amorphous, and the grainy, particulate nature of the
medium is a fundamental source of noise. Bertram's calculations have explored the
effects of two primary types of interaction between grains that cause noise: magneto-
static coupling and exchange coupling. The calculations resulted in recommenda-
tions for alloys and fabrication processes that would reduce noise from both sources.
The research enabled engineers at IBM's Almaden Research Center to design a disk
coating that packs a gigabit (one billion bits) of information onto a square inch,
which is 15-30 times current storage densities. Bertram has now turned his atten-
tion to calculations of other effects, including giant magnetoresistance, that can be
important in designing high-density disk media and disk recording and playback
heads. His codes model the process of recording bits in intricate detail; the process of
laying down a single bit of information takes several minutes to calculate on SDSC's
Cray Research C90.

Networking

One of the great successes of the NSF MetaCenter has been in providing the "high-
end pull" that has led to the creation and exponential evolution of the NSFnet. As a
result, the NSFnet backbone of 1995 has 3000 times the bandwidth of the backbone of
1986. The Centers have also prototyped the high performance local area networks
that are needed to feed into the national backbone as well as the next generation of
gigabit backbones. Security over networks is essential not only for industrial usage.

-9-

130

High Performance Computing Infrastructure and Accomplishments

but more and more for widespread citizen usage. Again, the MetaCenter has created
innovations such as dealing with industrial firewalls. The existence of the MetaCen-
ter network testbeds allows for new kinds of science to be attacked, perhaps best il-
lustrated by the rise of telemicroscopy, in which leading edge projects are being car-
ried out at each of the NSF Supercomputer Centers. With the national priority on
the Nn, the Centers are moving rapidly to expand their networking research to
community based Nil testbeds, including local healthcare, education, government,
and small business partners.

Evolution of NSFNET

The 56kbps connection between the NSF Centers, established in 1986, was the begin-
ning of the NSFnet. Based on the successes of ARPAnet and the TCP/IP protocol
within the computer science and Dept. of Defense communities, the NSFnet rapidly
grew to provide remote access to the NSF Supercomputer Centers by the creation of
regional and campus connections to the backbone. Although started by the pull from
the high end, the NSFnet soon began to provide ubiquitous connectivity to the aca-
demic research community for electronic mail, file transport, and remote login, as
well as supercomputer connectivity. These daily uses soon became indispensable to
the research community and the sustained exponential growth of the Internet took
off. The MetaCenter's industrial partner network was among the first in the corpo-
rate world to use NSFnet /Internet technology to connect corporations to the Internet
for the purposes of computational science. This was an important precursor for to-
day's rapid commercialization of the Internet.

By early 1995, the NSFnet will return to a high speed backbone connecting the Meta-
Center and some of the newly selected MetaCenter Regional Alliance members.
However, the bandwidth of the backbone will be 3000 times higher than that of the
original backbone (56 kbps). While some tend to think of the MetaCenter as focusing
on high performance computing only, it is useful to remember that computing
power of the fastest supercomputer processor in the program has grown by little
more than 100 times during the same period. Indeed, it is likely that the 155 mbps
vBNS will be upgraded to 622 mbps within two years. Even by the time the Centers
receive the first teraflop machines in 1997-98, realizing a factor of 1000 increase in
speed over 1985, the backbone will have grown by a factor of at least 25,000 fold in
bandwidth.

As part of an SDSC/UCSD collaboration, Kimberly Claffy recently completed a Ph.D.
dissertation that outlined a methodology for profiling Internet traffic flows at a vari-
ety of granularities. The methodologies and models developed as part of this traffic
characterization effort should prove very useful as the Internet evolves to an even
larger system in which the traffic composition needs to be understood, particularly
for planning future technology and capacity.

High Performance LANs

The center's program has also pioneered several transitions in local area and
metropolitan area networks both on site and on university campuses, acting as a pro-
totyping facility for other campuses who needed to know how to develop long range

10-

131

High Performance Computing Infrastructure and Accomplishments

networking plans for their campuses. In 1988, NCSA installed the first Ultranet Giga-
bit LAN networks with multiple supercomputers and demonstrated 480 Mbit/s be-
tween the CRAY-2 and Cray Y-MP supercomputers. In 1989, NCSA replaced tradi-
tional HYPERchannel backbone networks with the then-emerging 100 Mbit/s FDDI
standard. In 1991, PSC began its move to a HIPPI-based interconnect between its ma-
jor systems.

In 1993, NCSA, several industrial partners, and the UIUC Computer Science De-
partment established a local area ATM network testbed to help corporations gain
hands-on experience with ATM switches and interfaces. Insight from this ATM
testbed has already been used to develop long-range corporate network strategies for
J.P. Morgan, Phillips Petroleum, FMC Corporation, and United Technologies.

CTC was the first site to integrate ATM into a parallel supercomputer environment
on its IBM SPl in April 1994 and its IBM SP2 in July. ATM will be used for AFS-based
file service and other high speed transport needs, including distributed applications
and image transport.

Gigabit Testbeds

Since 1987, NCSA and the Uruversity of Illinois Computer Science Department have
worked with AT&T on the XUNET research network testbed with capacity that is one
step beyond what is available on the Internet. While the NSFnet has moved from 56
Kbs through 1.5 Mbps to 45 Mbs, XUNET has moved from 1.5 Mbs through 45 Mbs to
622 Mbs. In July 1993, XUNET was upgraded to 622 Mbs, the first network testbed to
interconnect ATM switches using 622 Mbs transmission technology over long (>50
miles) distance using pure optical fibers with in-line optical amplifiers.

PSC has worked with CMU's Computer Science Department in the Nectar Metropoli-
tan area gigabit testbed to develop new networking technology for very high-speed,
low-latency multi-machine interconnects and to develop the applications base which
can benefit from such technology. This work is fully collaborative with PSC's
ground-breaking systems and applications level work in heterogeneous systems. As a
result of testbed work, numerous applications can now run routinely between ad-
vanced machines at PSC's main hardware facility and those on the CMU campus, 15
miles distant, at speeds of up to 1 Gbit/s.

In partnership with NYNEX, Syracuse and Rome Laboratory, now extended to
Columbia, SUNY Stonybrook and Polytechnic Institute, CTC participated in building
a production-level ATM network focused on demonstrating research and commer-
cial applications. This network was demonstrated to the Governor of New York in
January 1994. NYNET is also a Nil testbed, involving outreach, medical applications,
video on demand, as described in a later section.

Various applications are being tested in the CASA testbed, in which SDSC is a major
partner. Besides the chemical reaction dynamics, led by Aron Kuppermarm of Cal-
tech and mentioned above in the section on superlinear speedup, there is a coupled
atmosphere /ocean model developed by the group led by Roberto Mechoso at UCLA.
Another is Calcrust, a project directed by JPL, which has used distributed heteroge-
neous computing over the CASA links to combine satellite imaging, seismic data.

132

High Performance Computing Infrastructure and Accomplishments

and surface topography in visualizing the foci of aftershocks of the 1992 Landers
earthquake in Southern California.

Secure Networks

In 1987 NCSA installed a 1.5 Mbit/s DS-1 connection to Eastman Kodak in Rochester,
New York, followed by another DS-1 connection to Amoco laboratories in Chicago
and Tulsa. By 1990 NCSA had connected over a dozen industrial laboratories to the
Internet using a combination of innovative security precautions. These included
various forms of "firewalls" which have now become commonplace on the Internet.

New Science Enabled by Networks— Telemicroscopy

The San Diego Microscopy and Imaging Resource (SDMIR), led by UCSD neurocy-
tologist Mark Ellisman, is an NIH-funded Research Resource centered on a new,
fully computerized Intermediate Voltage Electron Microscope (IVEM). The FVEM is
used to look at comparatively thick tissue sections (2-10 microns) and it has been
employed in studies of cortical neurons with and without symptoms of
Alzheimer's, in studies of another type of brain cell, called a Purkinje neuron, and
studies of cell membranes. A long-term collaboration between SDMIR and SDSC has
made the microscope usable interactively, over the Internet, coupled to the SDSC
Cray Research C-90.

Computational analysis and simulation is allowing biomedical researchers to study
and predict the activity of potential new drugs at the molecular level. CTC is working
jointly with Steven Ealick, et al., director of MacCHESS, a group using the Cornell
High Energy Synchrotron Source for Macromolecular Modeling. Using existing high-
speed connections between CTC and MacCHESS, the project is building the capability,
for the first time, for pharmaceutical companies and academic researchers to interact
dynamically with x-ray crystallographic analyses at the synchrotron, rather than dis-
covering long after the beam run that the sample was defective or the beam position-
ing non-optimal. While initially the researchers are using processors on the CTC SP
systems, ultimately a small IBM SP may be installed at the synchrotron site used for
dynamic analysis, with longer-scale simulation needs being met using the far larger
SP2 at tfie CTC.

PSC is working with the Center for Light Microscopy and Biotechnology, an NSF Sci-
ence and Technology Center at Carnegie Mellon University, to develop an Auto-
mated Interactive Microscope. This microscope will couple leading edge-microscopy
and high performance computing through high speed networks allowing the real-
time tagging of chemical reactants in the cell. It will open new research horizons in
biology by giving researchers the ability to control the release of chemically active
agents at critical moments in cell life, and to monitor the cell's subsequent develop-
ment.

The personal computer controlling a scanning tunneling microscope (STM) in the
Beckman Institute at UIUC used software integrated over a LAN with the NCSA
Convex C3880, TMC CM-5, and SGI graphics workstation to enable realtime imaging
and nanolithography of silicon surfaces in order to create novel quantum electronic
devices. Working with the laboratory's director, Joseph Lyding (Dept. of Electrical and

-12-

133

High Performance Computing Infrastructure and Accomplishments

Computing Engineering, UIUC and Beckman Institute) and inventor of a widely
used STM, NCSA staff members Rachael Brady and Clint Potter extended this
telemicroscope to the Internet and demonstrated the feasibility of using advanced
imaging instrumentation linked with advanced computing capabilities from any-
where in the world. The project was featured in the special issue of Research and
Development Magazine (Oct 25, 1993) on Winning in the 21st century.

Nil Testbeds

As a partner in Common Knowledge: Pittsburgh, an innovative project introducing
networking and computing into the entire Pittsburgh School District, PSC is working
with numerous partners, including Digital Equipment, Apple Computer and both
telephone (Bell of Pennsylvarua) and cable TV (TCI) companies, to create a prototypi-
cal, cost-effective approach to widespread use of advanced technology in public edu-
cation.

In collaboration with researchers and physicians at the Uruversity of Pittsburgh Med-
ical Center, the PSC is developing an Nil-based digital library of pathology images,
and the applications and software technology which wiU enhance the practice, teach-
ing and cost-effective delivery of pathology.

NCSA in collaboration with UIUC and the Champaign County Chamber of Com-
merce have been building CCnet, an Nil testbed during the last 18 months. Over 200
people from over 70 community organizations have been involved since April, 1993
in defining six major applications experiments in small business, health care, educa-
tion, government and community services, agribusiness, and geographic informa-
tion systems (GIS). The first of 20 multimegabit/s links into the community was es-
tablished with the Urbana Free Library in August 1994. All the high schools and a
number of small businesses are hooking on in September. NCSA is establishing a
large GIS server which will be available over CCnet to community projects. Partners
in CCnet include Time-Warner cable, Ameritech, DEC, Motorola, and potentially
MCI and AT&T.

NYNET, one of the gigabit testbeds, is also designated as an NE testbed providing
outreach, medical applications, video on demand to CTC and New York academic
and industrial partners.

InterNIC is the latest in an evolutionary line of support from the NSF for the use of
the Internet by the science, research, and education communities. The InterNIC pro-
vides three types of services: Information Services (provided through General Atom-
ics and the SDSC), Directory and Database Services, and Registration Services. Infor-
mation Services provides procedures for connecting to the Internet, pointers to re-
sources and tools available over the network, training seminars for new and experi-
enced users and up-to-date reports on new resources and activities on the Internet.
Several irmovative approaches to distributed services have been implemented, in-
cluding the InfoGuide, an on-line Internet information service. The Scout Report is
a weekly summary of Internet highlights which combines in one place the highlights
of new resource announcements and other news that occurred on the Internet dur-
ing the previous week. The InterNIC Reference Desk acts as the "NIC of first and last

-13-

134

High Performance Computing Infrastructure and Accomplishments

resort." The desk supports a variety of users answering "starter" questions from
novice users who are unfamiliar with the Internet as well as specialized questions
from intermediate and advanced users.

VISUALIZATION AND VIRTUAL REALITY

The NSF center's were instrumental in bringing the notion and tools of scientific vi-
sualization to the research commuruty in the 1980s. By combining advanced visual-
ization resources with simulation datasets created by remote users on the centers
program's high performance computers, new visualization paradigms for interpret-
ing numerical data were developed. This led scientists to consider visualization as an
intimate part of their computational toolkit. In addition, the centers worked closely
with the pre-existing computer graphics community to get them creating new tools
for scientists as well as for entertainment. Already by 1987, the staffs of the centers,
working with national users, were creating scientific visualizations so compelling
that they became regularly chosen to be part of the SIGGRAPH Film and Video
Show, the "academy awards" of the visualization industry. Today the centers visual-
ization staff and their allied visualization Centers are at the forefront of research into
how to turn virtual reality technologies into useful tools for scientific and engineer-
ing research.

Development of Scientific Visualization

From its inception, NCSA has worked with computer artists like Donna Cox (UIUC
Dept. of Art) and Dan Sandin (UIC School of Art and Design) and computer scientists
like Tom DeFanti (Dept. of Electrical Engineering and Computer Science, UIC) to cre-
ate cross disciplinary teams with end users in order to create new levels of scientific
visualizations. NCSA also hired a number of staff from leading companies in the
California entertainment industry to bring the software tools of special effects in
movies or TV commercials to the use of the scientific and engineering communities.
Initially, the NCSA visualization environment was built on Alliant shared-memory
multiprocessors using the Wavefront visualization software. As specialized graphics
hardware became available on Silicon Graphics systems, the NCSA visualization
environment migrated from the Alliant to Silicon Graphics systems. The scientific
visualizations created by NCSA s*^'ff have not only broken new ground for scientists
viewing their data, they have also won awards worldwide for aesthetic quality. NC-
SA's Renaissance Experimental Laboratory (REL), created by Donna Cox with a major
donation from Jim Clark (founder of Silicon Graphics) was the first advanced visual-
ization training facility in the centers and continues to support university courses in
geology, mathematics, graphics design, computer science, and other disciplines.

The SDSC has developed a variety of software tools that can be used to access and
connect existing visualization resources automatically. The goal has been to provide
researchers with training and access to tools that will support their research needs.
The tools include: 1) The SDSC Image Library, a collection of image manipulation
and conversion utility routines that can be embedded in existing software. An inter-
esting example of the use of this library are the image conversion modules devel-
oped by the International AVS Center, which quickly became the top 2 user modules
in their distribution; 2) the SDSC Image Tools, a collection of utilities based on the

-14

135

High Performance Computing Infrastructure and Accomplishments

Image Library, are software tools for reading, writing, and manipulating raster im-
ages. This toolset also allows researchers to convert file formats among over thirty
widely used graphics formats (e.g., from HDF to PICT). Now in its second release, it
runs on Cray Research, DEC, HP, IBM, SGI, and Sun Microsystems platforms. Over
4,000 sites worldwide have uploaded the Image Tools from SDSC's anonymous FTP
area; 3) vpr, a client-server visualization hardcopy utility, vpr takes advantage of the
Internet by allowing remote users to send images to local hardcopy devices. Popular
hardcopy devices that are connected to vpr include a variety of film recording de-
vices, a color paper plotter, and video recording; and 4) the SDSC Color Tutorial, a
SuperCard-based hypertext exploration in color theory for computer graphics. Exam-
ples show the different points that are being made, while hyperlinks allow the user
to jump to the most interesting references.

CTC Visualization staff developed Visual Programming Language for Animation
(VPLA), a program that easily integrates sound and image sequences into scientific
animations. VPLA can use rendered images from several standard visualization
packages including DataExplorer. As the national repository for Data Explorer soft-
ware and lead training site, the CTC works with faculty, industrial users and stu-
dents across the country in developing state-of-the art animations and images. In
addition, using DX, the CTC has spearheaded using visual programming languages
not only for visualizing, but for managing distributed applications running across
the centers program. For example, the CTC built a Data Explorer module allowing its
researchers to access the CMS at NCSA. The CTC was instrumental in IBM's agree-
ing to support DX across all major vendor platforms, including SGI.

PSC has concentrated its visualization efforts on the development of tools for re-
mote users. Its GPLOT software is in use at over 300 sites. Its automated animation
facility has enabled researchers to produce hundreds of videotapes without physi-
cally visiting the any specific center. It is now turning its efforts to develop such
tools embodying virtual reality.

Virtual Reality Impacts Industrial Design

In 1992 NCSA began a transition from the now-traditional workstation visualiza-
tion activities to virtual environments, with leadership provided by Caterpillar Inc.,
an NCSA Industrial Partner. Traditionally, translating electronic CAD blueprints
into full scale wooden models of new heavy earth moving equipment in order to
evaluate design changes required 6-9 months. Working with NCSA staff. Caterpillar
built up a VR laboratory in the UIUC Beckman Institute and networked the SGI
graphics workstations which create the VR images to their Peoria headquarters facil-
ities. Using a variety of VR viewing technologies, a number of design options al-
ready have been tested for new models of Caterpillar wheel loaders and backhoe
loaders that will be introduced by 1996. Design changes can now be made in less than
one month. Caterpillar design engineers Dave Stevenson and John Bettner received
the 1993 NCSA Industrial Grand Challenge Award for their innovative work. Media
coverage of this award reached over 200 media outlets.

-15-

136

High Performance Computing Infrastructure and Accomplishments

Development of Immersive Science Projects

The transition from stand-along workstation visualization to Nil distributed visual-
ization was emphasized at SIGGRAPH'92 in Chicago when NCSA collaborated with
the Ul-Chicago's Electronic Visualization Laboratory and dozens of science teams to
demonstrate wide area interactive visualization at Showcase. A new level of real-
ism in virtual reality was debuted there as well with the public showing of EVL's
Cave Automated Virtual Environment (CAVE), which provided complete immer-
sion in complex 3-D data sets at workstation levels of resolution. In 1993-1994, EVL,
NCSA, and Argonne organized a national call for proposals which resulted in over
60 EVL/NCSA computer science and visualization staff and graduate students help-
ing researchers from over 30 institutions in porting their applications into the
CAVE environment. For the first time, this included realtime coupling to parallel
supercomputers so that dynamic 3-D evolutions could be viewed immersively and
steered interactively. At SIGGRAPH 94, 8,000 attendees were able to directly experi-
ence these science projects. CTC developed a specific Cave Visualization on Macro-
molecular Modeling: the Structure of Acetylcholine Esterase; this visualization was
integral to the researcher's understanding of the molecule's activity. This applica-
tion runs not only on the CAVE's SGI workstations, but on CTC's IBM SP2 system as
well.

Virtual Reality over ATM networks

In a project IN 1994 with Rome Laboratory, demonstrated virtual reality techniques
over an ATM network between CTC SGI computers and Rome Laboratory. Research-
ing ATM technologies for real time applications and demonstrating software tools
for application steering important to molecular modeling, telemedicine and com-
mand and control.

The Sequoia 2000 Visualization Group at SDSC developed a prototype data visualiza-
tion system "Tecate" using virtual reality technology to address many of the issues
involved in exploring the informational content of networked data servers. Tecate
enables the browsing for data that resides in repositories managed by a database man-
agement system via user-interaction with graphical renditions of objects that repre-
sent data features.

Alpha Shapes, Biomolecules, and Cosmology

Alpha shapes, a form of geometric modeling developed by the 1993 Waterman
Award winner Herbert Edelsbruner (Dept. of Computer Science, UIUC) and NCSA
staff member Ping Fu, focuses on the formal definition, construction, and measure-
ment of shapes for any given point set in space. The discrete nature of the alpha
shape complex has computational advantages over any other known method which
can be exploited in computing surface area and volume of a space filling diagram and
in localizing and measuring voids. The latter is useful in studying water molecules
residing inside a protein. NCSA users have discovered other related applications of
alpha shapes by applying them to such diverse fields as adaptive grid generation,
medical image analysis, visualizing the structure of earthquake data, and the large-
scale structure of the universe.

-16-

137

High Performance Computing Infrastructure and Accomplishments

DIGITAL LIBRARIES AND INFOSERVERS

The National Information Infrastructure requires many software, computer, and
communications resources that were not traditionally thought to be part of high
performance computing. In particular, knowledge organization, location, and navi-
gating tools needed to be developed. The NSF Supercomputer Center staffs and their
associated universities have proven to be fertile ground for developing their new
tools. Perhaps the most spectacular success has been NCSA Mosaic, which in less
than 18 months has become the Internet knowledge browser of choice by over a mil-
lion users. The Mosaic/World Wide Web infrastructure has set off an exponential
growth in the number of decentralized authoring of information servers.

Digital Libraries

In 1989, NCSA as part of its XUNET application testbed proposed that multimedia
digital libraries would require gigabit networks in order to fully support high defini-
tion imagery and the coupling of large data sets with computing resources and geo-
graphically dispersed researchers. This resulted in research developments like DICE
(Distributed Collaboration Environment). Parallel efforts in providing researchers
with global information retrieval and display capabilities over existing environ-
ments combined collaboration with Bruce Schatz (then at U. Arizona) and his
Worm Community System, with internet-based designs of component client/server
architectures, like the World Wide Web. These approaches influenced the design of
the current grand challenge digital library prototype for accessing radio astronomy
images and data sets. NCSA built on the success of Internet access tools such as
NCSA Telnet, adapted this modified digital library paradigm to the Internet with
NCSA Mosaic. Today, the grand challenge image library uses NCSA Mosaic as it's
user interface, and Schatz has joined NCSA to head the recently awarded
NSF/ARPA Digital Libraries project, which combines a testbed based on the compo-
nent architecture with experiments in object-based designs.

Scalable Information Servers

The enormous success of NCSA Mosaic and CERN's WorldWideWeb has resulted
in explosive growth in the use of NCSA's WWW server. By the end of 1993, NC-
SA's server load had grown beyond the capabilities of any single server. This re-
sulted in the design of an innovative distributed scalable server architecture that in-
volved a modification of the Internet's Domain Name System software. By Sept.
1994, the NCSA WWW server was handling over 2 million connections per week.
NCSA's Hewlett-Packard workstation cluster based distributed information server
has now been duplicated at many WWW and FTP sites on the Internet and within
corporations. A number of corporations are presently working with NCSA on the
next generation of this distributed architecture.

The Rise of the MosaicAVWW Information Infrastructure

NCSA developed the Mosaic user interface software which provides point-and-click
access to the diverse information storage protocols of the Internet, such as World
Wide Web (WWW), Gopher, FTP, and WAIS. NCSA Mosaic establishes the neces-
sary connections, file transmissior^, decompression, launch of viewer programs, and

17-

138

High Performance Computing Infrastructure and Accomplishments

screen display of text, images, animations, or audio, in response to a single mouse
click from the user. NCSA Mosaic is available for Mac, Windows, and Unix comput-
ers for free to individual users, for government and educational use, and for internal
use within companies. Monthly download rates from the NCSA site alone are con-
sistently over 30,000. Although accurate estimates are difficult, it is widely felt that
over a million copies of NCSA Mosaic are in use. Further, commercial versions of
NCSA Mosaic are available. The principal licensee. Spyglass, Inc., has announced or-
ders for over five million copies of their enhanced version, with projections to
twenty million copies within a year. Use of NCSA Mosaic has increased WWW traf-
fic on the NSF backbone by over 10,000 fold since Jan. 1993. Overall WWW traffic in
August hit 1.3 Terabytes or 8 % of the total NSFnet backbone traffic, higher than
SMTP. Because of this, NCSA has become the second biggest Internet site in the
world in terms of traffic from its site. The NCSA Mosaic/WWW information infras-
tructure is allowing for an enormous growth in decentralized authoring of infos-
ervers throughout the world. In 1994, NCSA was given Infoworld's Publisher's In-
dustry Achievement Award.

DESKTOP Software

From the begirming, the NSF Supercomputer Centers provided focal points for
pulling together teams of computer scientists and software developers. Since the
history of the centers has greatly overlapped with the worldwide rise of the personal
computer and workstation, it is not surprising that the software developers focused
on creating easy-to-use software tools for the desktop machines themselves. These
tools have had a major influence on the usefulness of the supercomputer facilities
to the remote science and engineering community. The collaboration tools will
have a great impact on tying together the newly emerging electronic teams of scien-
tists made possible by the growth of the Internet.

Connectivity Tools

NCSA Telnet was a ground-breaking desktop application that provided access to the
emerging NSF Supercomputing Centers in the late '80s. Developed by the new
Workstation Tools Group (later the SDG) at NCSA, this brought full TCP connectiv-
ity to researchers using IBM and Macintosh systems, sigruficantly broadening the par-
ticipation base beyond Unix users, thereby introducing thousands to both the inter-
net and the NSF Centers Program. Continuously supported up to the present time,
these tools have also led to a spin-off company Intercon, headed by one of NCSA
Telnet developers.

Collaboration Tools

NCSA has supported a program of research and development on collaboration tech-
nology for science and engineering researchers for over 3 years. NCSA Collage, a tool
that runs across MSWindows, Mac, and XWindows systems, provides the capability
to carry on remote digital conferencing sessions between researchers. The first live
MetaCenter collaborative session using NCSA Collage was held in 1992 . Collage
combines many of the features of NCSA's communications and graphic data analysis
tools. NCSA also continues to innovate in asynchronous collaboration tools such as

-18

139

High Performance Computing Infrastructure and Accomplishments

asynchronous collaboration tools, hence the interest in annotation and workgroup
support capabilities in NCSA Mosaic. Current work focuses on choosing and combin-
ing the best of these synchronous and asynchronous capabilities in usable next-gen-
eration global collaboration tools for the scientific and educational communities.

Cornell University's CUSEEME video teleconferencing software, aimed at providing
video teleconferencing on low-end workstations, is in use at the NSF Centers and as
part of NYNET, NYSERNET and other organizations for routine use, including
medical projects between the CTC and the Cornell University Medical College in
NYC. This work, funded by NSF and Cornell itself, is freely distributed and runs on
Mac and PC platforms using inexpensive video equipment. Several national and in-
ternational collaborations have successfully utilized this software. The centers have
gained experience with traditional video teleconferencing systems, through its NSF-
funded system. It is now looking at packet video systems using the vBNS and other
network facilities. These systems will have the capability of moving the videocon-
ference from a set of specially equipped rooms to the desktop. An investigation is
now underway to develop the optimal system for CTC's requirements.

Graphics Tools

NCSA Image was the first scientific visualization tool developed for the desktop
viewing of supercomputing output in the program. It provided the research com-
munity Mac and Unix based visualization methods for analysis of huge data sets, as
well as creating some of the first client/ server tools which integrated remote desktop
workstations and personal computers with the center programs high performance
engines.

The SDSC Image Tools are software tools for reading, writing, and manipulating
raster images. This toolset also allows researchers to convert file formats among over
thirty widely used graphics formats (e.g., from HDF to PICT) and includes extensive C
library functionality for creating custom image-manipulation applications. Now in
its second release, it runs on Cray Research, DEC, HP, IBM, SGI, and Sun Microsys-
tems platforms. Binaries and sample source code are available in the public domain
by accessing SDSC's anonymous ftp area (ftp.sdsc.edu).

Scientist's Workbench

The Scientist's Workbench is an X and Motif-based software package developed at the
CTC. The main functions of the Scientist's Workbench are to bring together the tools
and software required by scientific researchers in a distributed computing environ-
ment, to provide a graphical interface to access those tools, and to provide the soft-
ware necessary to allow researchers to easily build their own graphical interfaces.
This tool has been used at most of the CTC's Smart Nodes (affiliates) by users and as
part of teaching environments for high performance computing, as well as at the
other centers and by companies and national labs developing "custom" program-
ming interfaces for their communities.

140

High Performance Computing Infrastructure and Accomplishments

Accon^plishments in Education and Outreach

Education

Familiarity with the tools of computation and visualization is quickly becoming a
sine qua non for both researchers and the public. The spread of access to these tools,
like access to the telephone and television before them, is a democratizing force in
itself: the world of the shut-in is opened up, the disadvantages of distance are mini-
mized, the exchange of techniques and knowledge is enhanced. Education, training,
and outreach are thus fundamental to the programs of the MetaCenter. Each mem-
ber of the centers program has developed educational programs targeted to a variety
of constituencies: university researchers, graduate students, undergraduates, educa-
tors at all levels, and K-12 students.

Researchers and Students

One- or two-day workshops are offered by centers program staff to researchers on site
and at associated institutions, covering introductions to the computational envi-
ronments, scientific visualization, and the optimization and parallelization of scien-
tific code. In addition, special workshops have been offered throughout the centers
program on the use and extension of computational and visualization techniques
specific to various disciplines (from biochemistry to lattice gauge theory). On the
campuses of centers program institutions and on other campuses, centers program
scientists and engineers are active teachers, either through regular academic ap-
pointments or as adjuncts, lecturers, seminar leaders, or teachers in extension divi-
sions.

Graduate students often receive fellowships or similar appointments at centers pro-
gram institutions, as their contributions may benefit a large academic research
community or the computational community generally. As an example, Ph.D. stu-
dent Kimberly Claffy (Computer Science, UCSD) recently completed her dissertation
on a flow-based measure of Internet traffic that she developed as a Junior Fellow at
SDSC. Dr. Claffy's technique is the first to permit traffic characterization on the basis
of a temporally and spatially flexible unit, and it is thus an enabling technology for
further advanced network research at SDSC and elsewhere.

The centers program has contributed to the research projects of hundreds of gradu-
ate students through stipends, access to resources, and relations with centers pro-
gram researchers. Each Center fosters collaborative research by muitidisciplinary,
multi-institutional teams of computer scientists, research scientists, and engineers;
postdoctoral research associates; and graduate students from the national and inter-
national community. These teams forge new approaches to previously insoluble re-
search problems, develop community codes, and host workshops and seminars to
transfer technology.

Supercomputer Centers Educational Activity Support Summary

Educational Activities

FY91

FY92

FY93

High School

20-

141

High Performance Computing Infrastructure and Accomplishments

Institutes

Attendees

128

131

121

Other K-12 Events

Attendees

715

1,370

1,985

Research Institutes

Attendees

262

377

390

Training Courses/ Workshops |

On Site Events

102

134

Attendees

1,414

1,773

1,929

Off Site Events

Attendees

295

622

104

Seminars/Colloquia

Events

138

114

132

Attendees

2,251

2,788

3,085

Academic Course Accounts

Monthly Newsletter Circu-
lation

234,986

247,692

165,176

Visitors

13,506

16,380

16392

For undergraduates, the Research Experiences for Undergraduates programs, funded
by NSF, bring in undergraduates to work for a summer or a school semester or quar-
ter on specific projects devised by centers program researchers and/or faculty advi-
sors. The projects are significant in their scope of computational science and in
many instances have resulted in presentations at meetings and publications. A spe-
cial project is CTC's Supercomputing Programs for Undergraduate Research (SPUR),
in which students apply to work on one of a selection of projects developed by Cor-
nell faculty in collaboration with CTC. One REU student at SDSC went on to win
the top prize in the Westinghouse Science Talent Search in 1991. Another devel-
oped a program to teach the use of the Braillewriter to blind students, which was
presented to the Commission on Equal Opportunity in Science and Engineering at
NSF (this student, herself blind, is now a successful computer scientist in Silicon
Valley). Undergraduate assistantships and internships are also available in the cen-
ters program. Undergraduate student programmers have worked on many research
problems including numerical weather prediction, the visualization of numerical
spacetimes, and social network analysis. They have developed numerous applica-
tions and utilities to improve the computational environment for MetaCenter re-
searchers. Students have also worked on library, visualization, and educational pro-
jects. The REU programs have been ongoing in various forms for more than five
years.

Outreach to Educators

One particularly effective approach to educating the next computational generation
is the training of teachers, and many centers program efforts have been devoted to
teacher training and curriculum development.

142

High Performance Computing Infrastructure and Accomplishments

Common Knowledge: Pittsburgh is a national pilot program developed by PSC, the
University of Pittsburgh, and the Pittsburgh Public Schools to institutionalize educa-
tional technologies within the Pittsburgh Public School District by having PSC im-
plement the network infrastructure and develop specific curriculum-based network
and computer applications. PSC's High School Initiative (1992-1994) involves stu-
dent/teacher teams using PSC facilities to develop computational tools for inclusion
in their schools' science or mathematics curriculum, with an emphasis on integrat-
ing high-performance computing into the curriculum and thus bridging the gap be-
tween textbook instruction and real world applications of scienre.

SuperQuest is a program involving centers program sites that brings teams of teach-
ers and students from selected high schools to summer institutes to develop compu-
tational and visualization projects that they work on throughout the following year.
In addition to educational workshop programs associated with SuperQuest, NCSA
has developed five interactive simulation programs now being tested in classrooms
across the country and around the world. These include GalaxSee, an N-body simu-
lator of galaxy formation and interaction; the Fractal Microscope, which enables the
exploration of self-similar patterns; SimSurface and SimElevator, simulated anneal-
ing programs; and LaplaceSeein', an electrostatic potential solver. Students can
change initial conditions and watch the simulation evolve as the parameter space is
explored.

SDSC's computational sciences curriculum coordinator, Kris Stewart (who is a pro-
fessor of mathematics at San Diego State University) has conducted summer work-
shops, funded by NSF and Cray Research, with faculty from primarily undergradu-
ate institutions to develop ways of incorporating high-performance computing into
the curriculum. Stewart uses the workshop materials in her own SDSU classes, Su-
percomputing for the Sciences and an Introduction to Computational Analysis.
SDSC is now halfway through a three-year, NSF-funded Supercomputer Teacher
Enhancement Program targeted to high-school teachers whose classes contain un-
derrepresented minorities.

Dr. Bruce Land, of the CTC, has developed an undergraduate course in scientific vi-
sualization and computer graphics, using the data flow block diagram capabilities of
IBM's Data Explorer software. This curriculum, lab exercises and the resulting stu-
dent projects have been shared with the larger educational community through the
CTC's Education and Training home page. Additionally, Prof. Steve Vavasis has de-
veloped an interdisciplinary course in scientific computing using high performance
computing for graduate and undergraduate students.

Over the past year, the CTC had established the Data Explorer repository, a full set of
tutorials for parallel computing on diverse platforms, a complete set of lecture notes
for use by educators as well as researchers, and a gateway to materials on the net-
work for secondary school science and mathematics education.

-22-

143

High Performance Computing Infrastructure and Accomplishments

of high-performance computation itself. The dissemination of trairung and curricu-
lum materials over the National Information Infrastructure is a major way in
which the successful pilot programs can be turned into a new class of educational re-
sources.
Outreach

Because advances in high-performance computing and communications (HPCC) are
driven by the needs of the practitioners with the most advanced problems, the cen-
ters program's scientific mission includes the construction of an extensive web of re-
lationships with research and development efforts in American industry and com-
merce. Collectively, the centers program's outreach programs represent a long
record of sustained collaboration among scientists, HPCC developers, and industrial
researchers. Another aspect of outreach is the effort to find and serve local and re-
gional needs of government, schools, and communities. Some aspects of these activ-
ities are discussed below.

Application of Scientific Computation and Visualization to Industrial Produc-
tion

Half of the partnerships between the individual centers and industry are collabora-
tions with major industrial firms. These include American Cyanamid, Amoco, Al-
coa, AT&T, Caterpillar, Corning, Dow Chemical, Eastman Kodak, Eli Lilly, FMC,
Gencorp, General Dynamics, Hughes Aircraft, IBM, JP Morgan, Martin-Marietta,
McDormell-Douglas, Merck Research Labs, Motorola, Parke-Davis, Philips
Petroleum, Schlumberger, USX, and Xerox.

In their original form, the partnerships represented the first introduction of large-
scale computation and visualization into the store of resources possessed by even
the largest of these Fortune 500 comparues. While the companies are for the most
part fully computerized now, the majority of these partnerships continue today be-
cause centers program expertise has been essential to the introduction of new ways
of employing the resources of supercomputing: the algorithms, visualization rou-
tines, and engineering codes are being combined in ways that result in such ad-
vances as high-end rapid prototyping of new products. As a result, for example, Eli
Lilly maintains its partnership although the company has purchased its own super-
computer — the useful interactions with centers program scientists, consultants, and
visualizers continue. In many of these arrangements, the industrial partner's re-
searchers are frequent visitors to the NSF centers, and centers program researchers
also visit the partner's installations.

Thus, while it is extremely important to Alcoa that it was able to produce an opti-
mum aluminum can, to Gencorp that it was able to design a better injection mold-
ing process, to McDonnell-Douglas that it could perform rapid airfoil analyses, to
American Cyanamid that it could reformulate soil enhancers, the sum of these
long-term relationships is important in another dimension as well. TheCenter out-
reach efforts are helping to revitalize American industry, making it more competi-
tive in an increasingly competitive world market.

■23-

144

High Performance Computing Infrastructure and Accomplishments

Impact on Vendors of High Performance Computing Equipment

The centers program has had a major impact on the vendors of major high perfor-
mance computing equipment. All Centers have taken early prototypes of machines,
have provided national access to the largest scale version of such machines, and
provided critical feedback to the vendors. Several have entered into strategic devel-
opment efforts v^ith the vendors. For example, PSC is a formal partner with Cray
Research in the development of applications for its massively parallel T3D. Cray
has also internalized some of PSC's file system developments. CTC played an inte-
gral role in IBM's re-entry into the High Performance Computing arena, as the first
customer for its IBM ES/3090 vector supercomputers and as a partner in the design
and development of its parallel FORTRAN products. CTC, through its director
Malvin H. Kalos, was a key influence in IBM's decision in 1991 to build the IBM SP
systems and ensured that IBM adopted a strategy that was scalable ultimately up to
the teraflops and down to the desktop.

SDSC has established a close collaboration with the Supercomputer Systems Divi-
sion (SSD) of Intel Corporation to develop systems software to support multi-user
systems, to serve as a test site for new operating system releases, and to improve the
stability of the Paragon system. SDSC staff have developed MACS, the Multi-user
Accounting and Control System, which Intel offers as part of the Paragon's standard
operating system software. This system includes a dynamic job mix scheduling algo-
rithm, a port of the Network Queuing System batch job submission software, and
CPU quota and accounting systems to control resources used by separate projects.

SDSC has also collaborated with Cray Research to develop support for multiple-user
systems on Cray systems. SDSC staff have developed a resource management system
that controls access to various resources on the system (CPU, memory, and disk) and
a dynamic job mix scheduler (DJMS) to dynamically adjust the workload for optimal
performance. SDSC has recently run a T3D emulator on the Cray C90 and is provid-
ing feedback on its performance. SDSC now plans to install and evaluate Cray's new
FDDI card, a fiber-optic high-speed network interface.

Digital Equipment Corporation recently awarded UCLA and SDSC an external re-
search grant to acquire nine Alpha 3000 model 400 workstations. The duster, which
has a peak speed of 1.2 Gflops and is connected at 100 Mbps via a Gigaswitch, will be
used primarily for climate studies led by Dr. Roberto Mechoso of UCLA. It will also
be available for scientific use and performance testing by the SDSC user community.
SDSC staff are collaborating with DEC to port the global climate model to the Alpha
cluster using DEC's High Performance Fortran compiler.

DEC is a major supporter of Project Sequoia 2000, a collaboration of scientists, com-
puter and information experts, government agencies, and industrial sponsors to de-
velop an information-management system for studying global climate change. The
Sequoia visualization group, centered at SDSC, has been developing a system that
will build on the strengths of existing hardware and software to support next-gener-
ation visualizations. Recently, the group collaborated with Kubota-Padfic Computer
Corporation to combine Kubota's Denali system with DEC Alpha machines for ad-

24-

145

High Performance Computing Infrastructure and Accomplishments

vanced 3D color graphics capabilities. Such collaborations have benefited DEC, Kub-
ota-Pacific, and the Sequoia 2000 project.

The National Storage Laboratory (NSL), a consortium that is developing next-gener-
ation high-speed storage devices, has selected the UniTree system as the production
archival storage system for all centers program sites. UniTree, which was originally
developed at the Lawrence Livermore National Laboratory, was commercialized l^
DISCOS, a spin-off of SDSC and a former division of SDSC's parent company. Gen-
eral Atomics. DISCOS was, in turn, sold to Open Vision, which continues to market
the product.

SDSC staff are collaborating with NSL, IBM, and Open Vision to implement the
NSL's base version of the UniTree archival storage system on the center's IBM
RS/6000 model 980 workstation. They are adding a new, more robust name server
developed by Lawrence Livermore National Laboratory for Open Vision, transaction
journaling (which allows reconstructing the database in case of catastrophic failure),
and adding and enhancing system administrator tools.

The integration of PSC's Multi-resident AFS into the NSL UniTree environment is
planned to provide user-friendly access for the centers program members into SD-
SC'S NSL archival storage system. The future integration of HIPPI-attached periph-
erals, including high-speed, high-density tape and high-speed RAID disk arrays, us-
ing third party data transfer is planned as a way to substantially increase both
archival system storage capacity and data transfer speed.

A prototype HPSS parallel I/O archival storage system is also planned for evaluation
as the follow-on to the NSL UruTree system. This system will support striping
across multiple high-speed peripherals to even further increase the speed of file
transfers.

Stimulation of New, Computationally Dependent Ventures

About a fourth of the industrial partnerships are with smaller and newer firms,
many of them leaders in biotechnology. Some are firms designing new pharmaceu-
ticals (e.g., Agouron Pharmaceuticals, Genentech), others develop and market the
software packages required for these enterprises. Biosym Technologies, for example,
is working with both CTC and SDSC to develop parallel versions of its popular Dis-
cover and Insight packages.

Outreach efforts of the centers program have resulted in actual spinoff ventures as
well. The commercialization of the software developed at individual centers is be-
ing undertaken by a number of companies. For example, NCSA Telnet has been
commercialized by Intercon, and Spyglass will release a package containing up-
graded versions of image tools and Mosaic. Some 20 companies have now licensed
NCSA Mosaic. CERFnet, a California wide-area network for Internet access has
pioneered in supplying access to library and other large databases; and
DISCOS/UniTree, a mass storage system, is in use at more than twenty major
computer sites. A new molecular modeling system, called Sculpt, developed at
SDSC, is being commercialized by a new company. Interactive Simulations. Sculpt
enables drag-and-drop molecular modeling in real time while preserving

-25

146

High Performance Computing Infrastructure and Accomplishments

minimum-energy constraints; its output was featured on the cover of Science last
May.

Development of Nationally Valuable Reservoirs of Skill

About a fourth of the partnerships between the centers and industry are collabora-
tions with manufacturers of high-performance computational, networking,
telecommunications, and visualization equipment. Of particular interest here are
the several partnerships funded by the NSF and ARPA through CNRI to construct
and test the "gigabit testbeds," prototypes of the connectivity that will be required for
the future International Information Infrastructiire. Both academic and industrial
research groups are developing the codes to test the connections, even as manufac-
turers develop the connections themselves, and the experts assembled in the centers
program supply links in the form of specifications and software.

Community Service

Local and regional outreach efforts range from the tours given at all centers program
installations through the hosting of visits by national, regional, and local officials
and commissions, to the kinds of full-scale partnerships mentioned above. The
NCSA relationship with the Champaign County Chamber of Commerce has re-
sulted in the formation of a nonprofit public network, CCnet, which is already bene-
fiting the Chamber itself as well as local schools. Plans are in the works with Time-
Warner to start pilot tests of the use of public-access cable for a data highway.

SDSC is working with the City of San Diego on plans to connect all units of dty gov-
ernment, including a high-technology resource center to be developed with De-
partment of Commerce funding that will connect local industry (with a lot of de-
fense reconversion efforts) to business and computational resources, including
SDSC itself.

PSC is exploring extension of the technology it developed for its Common Knowl-
edge:Pittsburgh K-12 project to embrace major units of city government.

Outreach is also represented by the publications programs of the centers program,
the production of scientific videos and /or multimedia CD-ROMs, and a collabora-
tive program for maintaining a lively and informative presence on World-Wide
Web servers, which will make information on the programs easily accessible over
the Nn.

26-

147

High Performance Computing Infrastructure and Accomplishments

Important Science and Engineering Accomplishments

Summaries of computationally interesting problems in the NSF Centers Program by
the national science and engineering communities:

QUANTUM PHYSICS AND MATERIALS

The great disparity between nuclear, atomic, or molecular scales and macroscopic
material scales, implies that vast computing resources are needed to attempt to pre-
dict the characteristics of bulk matter from fundamental laws of physics. Therefore,
it is not surprising that since the begirming of the NSF Centers program this area of
science has brought us some of the largest users of supercomputers. Materials scien-
tists have often been among the first group of researchers to try out new architec-
tures that promise even higher computational speeds.

Below are outlined some outstanding examples of studying properties of bulk mat-
ter from extreme conditions, such as occur in nuclear collisions, the early universe,
or in the core of Jupiter; new materials such as nanotubes and high temperature su-
perconductors; and more practical materials used today such as magnetic material
and glass.

Phase Transition in QCD

The MIMD Lattice Calculations Collaboration (MILC) is attacking the Grand Chal-
lenge problem of "the origins of mass." Their objective is to use the theory of the
forces governing what are called the "strong interactions" of elementary particles
(quarks and gluons) to calculate the observed masses and interactions of the particles
that are made out of them: the hadrons, which include the familiar proton and neu-
tron. The theory is called quantum chromodynamics (QCD), and its numerical in-
carnation is called "lattice gauge theory," because the quarks and gluons are repre-
sented on a four-dimensional space-time lattice. They have published numerous
studies of the mass spectrum of the hadrons; the transition between ordinary matter
and the quark-gluon plasma, which is important in the study of the conditions of
the early uruverse; and the decays of hadrons via weak interactions. A number of
investigators, coordinated by Robert Sugar (Dept. of Physics, UCSB), are engaged in
this project including: Claude Bernard (Washington Uruv.), Thomas A. DeGrand
(Univ. of Colorado), Carleton DeTar (Univ. of Utah), Steven Gottlieb and Alexander
Krasnitz (Indiana Univ.), Douglas Toussaint (Univ. of Arizona), Julius Kuti (UCSD).
The consortium has used large allocations of time on a wide range of MetaCenter
computational facilities including: Intel Paragon (SDSC), TMC CM-5 (NCSA), clus-
tered IBM RS/6000S under PVM (CTC, NCSA), Cray Research C-90 (SDSC and PSC)

Phase Transitions of Solid Hydrogen

Calculations by Natalie, Martin and Ceperley (Dept. of Physics UIUC, NCSA), carried
out on the CRAY Y-MP at NCSA have established the series of crystalline phase
transitions of hydrogen as it is compressed to several million atmospheres of pres-
sure, such as found in the interior of the giant planets. Since Wigner and Hunting-
ton in 1935 pointed out that a transformation from a molecular to atomic state is in-
evitable at high pressure, there have been extensive speculations on when and how

27-

148

High Performance Computing Infrastructure and Accomplishments

this transformation would take place. The recent development of the diamond
anvil technique have allowed experiments to be performed at pressure slightly
lower than the atomic transition. Those experiments confirmed the existence of an
molecular orientation transition which had been earlier computationally predicted
by Ceperley at l.SMbar. Extensive and highly accurate quantum Monte Carlo calcu-
lations on a variety of crystal structures now predict that the metallic transition will
take place from a distorted molecular hexagonal structirre into an atomic diamond
lattice. For this and other pioneering science, David Ceperley was awarded the fifth
Eugene Feenberg Memorial Silver Medal in 1994. David's PhD advisor and the third
Feenberg awardee is Mai Kalos, Director of the CTC, himself a major user of several
MetaCenter supercomputers.

Prediction of new Nanomaterials

Marvin L. Cohen (NAS) and Steven G. Louie (Dept. of Physics, UC Berkeley) have
used MetaCenter computational resources (SDSC, PSC, NCSA Cray Research Y-MP
and C90) to make numerous advances in computational materials science. Most re-
cently, they have used both first-principles and tight-binding codes to examine the
properties of carbon nanotubes and nanotubes composed of boron, carbon, and ni-
trogen. Carbon nanotubes-essentially rolled microsheets of graphite-are well
known, thanks to the work of Sumio lijima and colleagues at NEC. They have di-
ameters on the order of 1-20 nm and are producible in the same carbon arc chambers
used to produce fullerenes (also called "buckyballs"- assemblages of 60 or more car-
bon atoms in cage-like structures). They have interesting capillarity and electronic
properties. Cohen, Louie and their colleagues have predicted the structure and
properties of nanotubes made of boron nitride, which appear to be more stable and
controllable in terms of their electronic properties. They have also predicted nan-
otubes of BC_2N, a boron-carbon-nitrogen compound, whose electroruc properties
are even more interesting: they should behave like nanoscale induction coils. Most
exciting, the structures that were predicted computationally are now being produced
experimentally in the lab of Alex Zettl at Berkeley, where their electronic properties
can be confirmed.

Theory of High Temperature Superconductors

The Nobel Prize in Physics in 1987 was for the discovery of a new class of high tem-
perature superconductors. Thousands of research papers have been written about
these unique materials, but the battle is still raging over the fundamental mecha-
nism that causes the superconducting transition at Tc ~ 90K for the cuprate oxides
such as YBA2CU3O7. David Pines (NAS and first Feenberg Medalist) and Philippe
Monthoux (Department of Physics, UIUC) used the NCSA Cray Research Y-MP to
carry out a strong coupling (Eliashberg) calculation of the normal state properties
and Tc for the model experiment-based magnetic interaction between quasiparticles.
They found that when the full structure of the quasiparticle interaction is taken into
account, a superconducting transition into a d-wave planar pairing state occurs at Tc
~ 90K for comparatively modest values of the coupling constant. Although still an
area of active research, this computation lends credibility to the model that it is the
coupling of planar quasiparticles to the experimentally measured planar electronic

-28-

149

High Performance Computing Infrastructure and Accomplishments

spin fluctuation excitations which determines the normal state properties (which
they show acts like a nearly antiferromagnetic Fermi liquid) and makes possible
high temperature superconductivity.

Magnetic Materials

James Sethna (Laboratory of Atomic and Solid State Physics, Cornell Univ.) uses
CTC parallel supercomputers to study the dynamics of disorder-driven first-order
phase transformations, including 3-D numerical simulations of hysteresis loops. He
is developing scalable parallel algorithms for systems of size N = 20003. Sethna's
work enables the prediction of phase transitions with critical fluctuations, the simu-
lation of orders-of-magnitude larger systems to explore critical phenomena, and de-
tailed computational studies in materials science as applied to magnetic storage me-
dia, metallurgical phase transformations, and gases adsorbed on surfaces.

Understanding Glass

For ab initio dynamical calculations to be useful for real materials in an industrial
setting, they must be able to deal with ensembles of thousands of atoms for dynami-
cal effects modeled over microseconds. Significant algorithmic developments made
jointly at Corning, Inc. and the CTC, coupled with the much increased capability of
the CTC's IBM SP2 system, allow this threshold to be crossed for the first time. Post-
doctoral Fellow Stefan Goedecker, hired jointly by Corning, Inc. and CTC, has devel-
oped new extremely fast ways of doing tight-binding which he can parameterize
with the ab initio codes. This is the only approach currently known that will handle
thousands of atoms for millions of time steps, bringing the researchers close to ob-
serving many of the mysteries involved in glass chemistry which have been not
well understood for over 2000 years.

Biology AND Medicine

Living creatures exhibit some of the greatest complexity found in nature. Therefore,
supercomputers have made possible unprecedented opportunities to explore these
complexities based on the fundamental advances made in biological research of the
last fifty years. These activities include: inverting the data from x-ray crystallography
experiments to obtain the molecular structure of macromolecules; learning how to
use artificial intelligence to fold polypeptide chains, determined from genetic se-
quencing, into the three-dimensional proteins; and determining the function of
proteins by studying their dynamic properties, as well as how they interact with each
other or with the DNA backbone from whence they were created.

These insights are beginning to make significant impacts on medicine and plant and
animal biology. New fields of computational science, such as molecular neuro-
sciences, are being enabled by academic access to MetaCenter computing and visual-
ization resources and staff. Corporations are using supercomputers and advanced
visualization techniques in collaboration with the NSF MetaCenter to create new
drugs to fight human diseases such as asthma. New ir\sights into economically
valuable bioproducts are being gained, for instance, by combining molecular and
medical imaging techniques to create "virtual spiders" which can be digitally dis-
sected to understand the production of silk. Finally, high performance computers

-29-

oc—m D r^

150

High Performance Computing Infrastructure and Accomplishments

are just becoming powerful enough that some dedicated researchers are able to pro-
gram mathematical models of realistic organ dynamics, such as the human heart.

Crystallography

Herbert Hauptman (Medical Foundation of Buffalo, Inc.) won a Nobel Prize in 1985
for development of the "direct method" of protein structure determination from X-
ray crystallographic data. In a collaboration with Russ Miller (State University of
New York at Buffalo), these researchers have developed a numerical approach that
extends the "direct method" of determining molecular structure from X-ray crystal-
lographic data to larger molecules, beyond its present limit of about 100 atoms. The
algorithm they have developed, called "Shake-and-Bake," runs on a number of
computing platforms (PSC CM-2 and Cray T3D, NCSA and PSC CM-5) and has
proven itself effective in more than 20 cases at accurately determining the structure
of proteins that have taken as long as 10 years by existing methods, reducing the
time to a matter of hours.

Folding Proteins using Artificial Intelligence

One of the most pressing problems in molecular biology is how to determine the
folding and 3-D structure of a protein, given its sequence. Peter Wolynes (NAS), Zan
Schulten, and coworkers (Dept. of Chemistry, UIUC) have developed a novel ap-
proach to this classic problem using elements from the theory of spin glasses, asso-
ciative memory models, and neural networks. Spin glass theory provides a frame-
work for understanding the cooperative nature of the folding transition and the
qualitative nature of the phase diagram describing the thermodynamics of proteins.
Wolynes et al. developed simulation codes, based on associative memory Hamilto-
nians, and characterized their phase diagrams semi-quantitatively. These Hamilto-
nians are based on an energy function which correlates the sequence of the protein
to be folded with those of proteins whose structure is known. They were introduced
several years ago by Wolynes and coworkers as polymer analogues of the Hopfield
neural nets. Their work, carried out on NCSA's Cray-2 and CRAY Y-MP, shows that
even primitive associative memory Hamiltonians can recognize protein structures
from sequences that are only moderately related to those already existing in the
database. These procedures are somewhat similar in effectiveness to the rule-based
homology modeling.

Protein Kinase solution

The Computational Center for Macromolecular Structure (CCMS), founded in 1990,
is an NSF-fundcd joint project of UCSD, SDSC, and The Scripps Research Institute,
with collaborators from all over the country. The center made headlines in 1991
when a group led by Susan Taylor (Dept. of Chemistry, UCSB), one of the principal
investigators of CCMS published the three-dimensional structure of the catalytic
unit of cyclic-AMP-dependent protein kinase, or cAPK. This was the first kinase
structure to be solved. The solution was achieved by a combination of computa-
tional methods, including refinement on the SDSC CRAY Y-MP using the program
XPLOR, developed by Axel Bruenger of Yale University. Most important to the solu-
tion was the ability of the scientists to study stereo visualizations of the structure on

■30

151

High Performance Computing Infrastructure and Accomplishments

a large screen at SDSC, so scientists from every discipline within the group could
contribute their insight to a collective determination of the structure. Kinases play
important messenger roles in cell metabolism, and hundreds of such compounds
have been identified and sequenced. Because sequences are homologous in long
stretches, the solution for cAPK is proving extremely valuable as a template for
modeling and deriving the structure of other kinases. Taylor and her group have
collaborated with several other groups since in modeling proposed solutions for
other kinases, including those known to have carcinogenic properties or to be in-
volved in other disease processes. In all of these studies, computation and visualiza-
tion have played an important role. Solutions for various kinases can lead to the de-
sign of inhibitors to prevent the enzymes from acting to produce diseases. The work
won the Forefronts of Large-Scale Computation Award presented at Supercomput-
ing '93.

Molecular Neuroscience-Serotonin

A number of cardiovascular and psychiatric diseases are currently treated with drugs
that act on the neurotransmitter serotonin and its receptors. The cellular receptor
for serotonin is a gatekeeper molecule that recognizes and binds the serotonin and
then transmits the signal to the cell by binding to a special class of transducers: the
G-proteins. Using the CTC's ES/9000, Dr. Harel Weinstein, chairman of the Depart-
ment of Physiology and Biophysics at Mount Sinai Medical Center, made a break-
through in modeling the serotonin receptor. His breakthrough came from modeling
the structural changes that occur in the serotonin receptor when it binds to a ligand
and to the G-protein, causing it to carry out its function. This research shows how
G-proteins can be switched on by structural changes in specific regions of the recep-
tor molecule and is expected to lead to the development of more effective drugs,
specific ligands aimed at the regions where the structural change takes place. Wein-
stein believes that his work may be applied more broadly to other receptor
molecules, including all neurotransmitters, that communicate with cells via G-pro-
teins. If Weinstein can demonstrate a common mechanism of response in these re-
ceptors, he will have a new typ)e of molecular approach to treating a vast range of
diseases. Weinstein has also used the Cray Research C-90 and Intel Paragon at SDSC.

Molecular Neuroscience-Acetylcholinesterase

A collaboration between Michael Gilson, T.P. Straatsma, and Andrew McCammon
(Dept. of Chemistry, University of Houston), Daniel RipoU (Research Associate,
CTC), Carlos Faerman (Dept. of Molecular and Cell Biology, Cornell Univ.), Paul
Axelsen (Dept. of Pharmacology, University of Pennsylvania School of Medicine),
and Israel Silman and Joel Sussman (Weizmann Institute of Science, Israel) has
used molecular dynamics algorithms to investigate the rapid activity of the enzyme
acetylcholinesterase (AChE). The enzyme breaks down the neurotransmitter acetyl-
choline diffused across nerve cell synaptic gaps. Its three-dimensional crystal struc-
ture was solved by Joel Sussman and colleagues at the Weizmann Institute in Re-
hovot, Israel, several years ago. That structure showed the active site to be a long,
narrow channel — too narrow to deal rapidly with the job of dissociating acetyl-
choline into choline and an acetate ion. Yet it is known that AChE acts very rapidly.

•31-

152

High Performance Computing Infrastructure and Accomplishments

no doubt because of aeons of evolutionary pressure to optimize the response of the
nervous system in all organisms. Molecular dynamics calculations performed on
the MetaCenter's Cray Research C-90 (SDSC and PSC), TMC CM-5 (NCSA), and
Kendall Square KSR-1 (CTC) showed that there was also a "back door" to the active
site, that might open to facilitate the exit of the acetate ion from the site. A study of
the electromagnetic fields of acetylcholinesterase with the back door closed and then
open supplied confirming evidence. Since inhibitors of AChE are important medi-
cations for myasthenia gravis, glaucoma, and Alzheimers's disease, this new insight
may lead to more effective pharmaceutical agents to fight these diseases. This work
was the cover story of the March 4, 1994 issue of Science magazine.

Kinking DNA

John M. Rosenberg (University of Pittsburgh) used the PSC Cray Research C-90 vec-
tor supercomputer to determine how a protein identifies and interacts with specific
sites of DNA — a fundamental biological process called "protein-DNA recognition,"
which is related to many disease processes and is also a vital tool in the biotechnol-
ogy industry. Rosenberg's molecular dynamics simulations have refined the struc-
ture of an important protein, Eco Rl endonuclease, used in DNA cloning, and they
have resulted in a clear understanding of a "kink" in the DNA backbone that results
when Eco Rl endonuclease binds with DNA. Rosenberg won the 1991 Forefronts of
Large-Scale Computation award for this research, and his work was cited in the 1993
Computerworld Smithsonian award for science given to the PSC.

Antibody-Antigen Docking

A collaboration among computer scientists Michael Hoist and Faisal Saied (Dept. of
Computer Science, UIUC) and two biologists Richard Kozack and Shankar Subra-
maniam (Dept. of Physiology and Biophysics, UIUC and Beckman Institute/NCSA)
has been able to solve for the first time the complete nonlinear Poisson-Boltzmann
equation, which is the fundamental equation of macromolecular electrostatics. A
method based on multigrid-inexact Newton algorithms has been developed and
large memory applications run in parallel on the NCSA Convex C3 show that this
has profound consequences for protein structure, enzyme mechanisms and protein
design. Coupling this new approach with a Brownian dynamics method, the largest
simulation ever of an encounter between two proteins, an antibody and an antigen,
has been carried out using the NCSA CM-5 and SGI Challenge. This simulation for
the first time is able to give rate constants for association of proteins that is compa-
rable to experimental measurements. The results of the electrostatics work was the
cover story of the March 1994 issue of Proteins: Structure, Function, and Genetics.

Tuning Biomolecules to Fight Asthma

Over the last 20 years, the number of asthma cases has almost tripled in the U.S.
David Herron, senior research scientist at Eli Lilly and Company, is searching for
new drugs that will inhibit the action of leukotrienes, inflammatory agents released
by several types of cells in the lungs, which cause the lungs to stiffen and become ir-
ritated. Several gigabytes of data from molecular dynamics of three key
leukotrienes, run on NCSA's and Lilly's Cray-2 supercomputer were analyzed in a

-32-

153

High Performance Computing Infrastructure and Accomplishments

lengthy scientific visualization created with NCSA staff. Using the animations as a
guide, Lilly asthma researchers synthesized highly active antagonists against the
leukotrienes. Some have been tested in asthma sufferers and found to be effective
medically. For this work, Herron was a co-recipient of the first NCSA Industrial
Grand Challenge Award.

Virtual Spider and Artificial Silk

Biophysicist Lynn Jelinkski, director of Cornell University's Biotechnology Center
for Advanced Technology (CAT) is combining medical imaging techniques with the
state-of-the-art computer visualization resources of the CTC to study the molecular
structiire of the strongest silk of the golden orb weaver spider and its transformation
from a viscous fluid into the extremely strong crystalline fiber which has the poten-
tial to replace manmade fibers, such as nylon, manufactured from petrochemicals.
Jelinski has devised a way to create a 3-D "computer spider" by compiling stacks of
the 2-D MRI images using the IBM POWER Visualization System, one of the high-
performance computing resources of the CTC. Each image contains over 100,000
pixels. Hundreds of images are combined to construct the 3-D simulated spider.
Once in hand, this virtual spider can be dissected by computer to describe the
anatomy of the glandular system and to provide the physical processing informa-
tion Jelinski seeks. This understanding, coupled with molecular-level studies of the
amino acids that make up the web silk polymer, may aid in geneticaUy engineering
plants to produce fibers as strong as those produced by the spider. Jelinski's work
blazes a path toward the development of a new class textiles with superior strength
at the same time that it promises fundamental insight into the mystery of the spi-
der's web.

Heart Modeling

Charles S. Peskin and David M. McQueen (Courant Institute, New York University)
have developed over the last decade a fully functioning three-dimensional model of
the heart, its valves and nearby major vessels. This computational model will make
it possible to study questions about normal and diseased heart function that are dif-
ficult or impossible to address through animal and clinical studies. The complexity
of the heart model is so great that a single heartbeat requires a 150 hour run on the
PSC Cray Research C90 and could not have been run without the very large memory
of the C90. This research won the 1994 Computerworld Smithsonian award for
Breakthrough Computational Science. Peskin was awarded a MacArthur Prize Fel-
lowship in 1983.

Engineering

Man-made devices have become so complex that researchers in both academia and
industry have tvirned to supercomputers in order to be able to analyze and modify
accurate models in ways which complement the traditional experimental methods.
Such easily accessible high performance computers enable academic engineers to
study the brittleness of new types of steel, to improve bone transplants, or to reduce
drag of flows over surfaces using riblets. Industrial partners of the individual super-
computer centers within the MetaCenter are using computational facilities more

•33-

154

High Performance Computing Infrastructure and Accomplishments

advanced than they have access to internally to improve industrial processes such as
in metal forming. Better consumer products such as leakproof diapers, or more effi-
cient airplanes are being designed. Even State agencies are able to use the MetaCen-
ter facilities to improve traffic safety or find better ways to use recycled materials.
Some 70 corporations have taken advantage of the MetaCenter industrial programs
to improve their competitiveness.

Ultra-high-strength Steels

Gregory B. Olson and Arthur J. Freeman (Northwestern University) use computer
modeling to design ultra-high strength steel for weight-critical applications such as
naval aircraft landing gear, high-performance race cars, and bearings in the main
engine turbo pumps of the space shuttle. In recent supercomputer modeling on the
PSC Cray C90, applying quantum mechanical calculations to the structure of steel,
they have explained the molecular mechanisms that give rise to impurity-induced
embrittlement in steel, work which is expected to lead to steel that will not shatter
in frigid conditions. This work was reported in the July 15, 1994 Science. Freeman
and his group have been users of NSF supercomputers since the founding of the
program on a wide range of problems in materials sciences. In recognition of his pi-
oneering work in computational materials research. Freeman received the first Ma-
terials Research Society Medal and the first Award in Magnetism from the lUPAP.

Continuous Casting of Steel

Achilles Vassilicos (U.S. Steel Technical Center) models the flow of molten steel in a
continuous-casting "tundish" on the PSC Cray Research C90, resulting in improved
process control over the quality of steel. By more acciirately predicting the precise
metallurgical composition of the continuous-casting output "slabs," U.S. Steel re-
duces waste steel and the amount of inventory it must keep on-hand, resulting in
substantial cost savings.

Beverage can design

Three-dimensional stress modeling of aluminum beverage cans on the PSC C90 by
Robert E. Dick and Andrew B. Trageser (ALCOA Laboratories) has greatly reduced
the expense of developing a new can design that will meet customer specifications
for strength and appearance. By relying less on costly, time-consuming prototype
testing, ALCOA engineers estimate a cost savings of $100,000 or more per can design.
This research has been described in articles in Discover (March 1991), Business Week
(Oct. 8, 1990) and in Science Qune 23, 1989).

Designing a Leakproof Diaper

Designing effective and comfortable disposable infant diapers requires greater un-
derstanding of the function of the diaper components, such as the cellulosic fluff
and the superabsorbent polymer particles-and the effect of variations of parameters
related to these components. Dow Chemical Company, has done extensive experi-
mental testing, evaluation, and computer modeling that has contributed to a faster
developmental process and shortening the time for new product introduction. The
innovative Dow design was evaluated using a computer model run on NCSA's

-34-

155

High Performance Computing Infrastructure and Accomplishments

CRAY-2 system. Three separate time-dependent processes are modeled. The first, a
fast spreading process, involves the insult on the pad by a quantity of liquid (urine)
which is transported through the pad by wicking. During this process the fluff pad
collapses as it becomes wet. The second, called the imbibition process, models the
swelling of the superabsorbent polymer particles and their uptake of liquid from the
cellulose fluff. During this process, the fluff expands again. The slowest-and final-
process tracks the redistribution of liquid in the fluff pad as the saturation of the
fluff adjacent to the superabsorbents changes. The overall model was compared to a
magnetic resonance imaging experiment, which provides a three-dimensional im-
age of the water distribution in a diaper, and was shown to give comparable results
to the final steady-state values. Optimization of these processes is leading to an im-
proved, quality diaper.

Bone Transplant Bioengineering

Dean Taylor and Donald Bartel (Dept. of Mechanical and Aerospace Engineering,
Cornell University) have been able to investigate bone-implant systems across a
wide range of design parameters by using high performance parallel computing
(including the IBM SPl) and visualization resources at CTC. Their long-term re-
search has produced models of the stresses placed on normal bones and the artificial
components of a hip joint — these models have led to custom-designed prostheses
and reduced prosthesis replacement surgery.

Improving Performance with Riblets

George Em Karniadakis^ and his group at Brown University are using the SDSC In-
tel Paragon to explain drag reduction in turbulent flow by means of "riblets," paral-
lel grooves on the surface of an object moving through the flow. Such grooves are
used on aircraft, in pipelines, and on radng cars and sleds, to improve performance.
The group also models flows in micro-electromechanical systems used in surgery
and other complex applications, where the molecules of the fluid are not much
smaller than the channels in which they flow. Both projects are resulting in new
ways to optimize the performance of broad classes of machinery.

Designing Better Aircraft

Dino Roman, John Vassberg, and Tom Gruschus of McDonnell Douglas are using
SDSC's Cray C-90 for a computational fluid dynamics simulation of an aircraft in
flight and to visualize the results using FAST (Flow Analysis Software Toolkit) on a
Silicon Graphics IRIS workstation. Tracer particles are released into the flow field in
front of the aircraft and allowed to follow the streamlines around the vehicle. A cut-
ting plane through the 3D volume of data is placed to intersect the aircraft fuselage
and wings. The aerospace industry relies on computational fluid dynamics-the
simulation of air or fluid flow-to design, develop, and test new aeronautical config-
urations. This process enables companies to test new models quickly to select candi-

' Incidentally, Prof. Kamiadakis is the current chair of the joint NCSA/PSC National Peer Review
Board, another example of the tight links between members of the MetaCenter and the scientific
community.

-35-

156

High Performance Computing Infrastructure and Accomplishments

dates for wind-tunnel testing. Such methods used in aircraft design and manufac-
ture can give American companies a technological edge in the global market.

Crash Testing Street Signs

California State Department of Transportation (Caltrans) engineer Payam Rowhani
and SDSC engineer Chuck Charman generated a computerized model of a crash-test
vehicle, called a "bogie," on the SDSC Cray Research Y-MP for simulations of test
crashes with sign and lighting supports. They fine-tuned the bogie front-end design
with the computer model to minimize the number of validation tests necessary at
the Federal Outdoor Impact Laboratory, operated by the Federal Highway Adminis-
tration. In a second application, Charman is working with Caltrans engineers
William Nokes and Dario Perdomo, who are designing pavement using structural
modeling techniques. They are researching the use of new and recycled materials
and these materials' response to different axle and tire configurations. They are us-
ing the supercomputer and the visualization facilities to explore new truck suspen-
sion systems, and new tires and heavier loadings, innovative pavement structures
with recycled materials and rubber and polymer-modified binders. The results of
this research are expected to lead to significant cost savings in the design, construc-
tion, maintenance, and rehabilitation of pavement structures.

Earth Sciences and the environment

From understanding the motions of the Earth's convective mantle to daily compu-
tation of air pollution levels in southern California, the resources of the NSF Meta-
Center are being used to compute and visualize the complexity of the natural world
around us. The US Army is working with academics to determine how they can
practice tank maneuvers without endangering the breeding habits of the sage
grouse. Pollution, whether underground or in the air, is a difficult coupling of
chemical reactions and flow dynamics which must be understood in detail if correc-
tive measures are to be efficacious. High performance computers also act as time
machines, allowing for faster-than-realtime computation of severe storms. Finally,
to improve global weather or climate forecasts, supercomputers allow researchers to
zero in on the critical coupling physics of such processes as mixing at the air/ocean
interface.

Detoxification of Ground Water

Christine Shoemaker (Dept. of Civil and Environmental Engineering, Cornell Uni-
versity) has been a pioneer user of the scalable IBM SP machines at the CTC for the
development of numerically efficient supercomputer algorithms for optimal con-
trol of dynamical systems and the application of these techniques to detoxification of
contaminated groundwater. Her efforts are leading to methods of determining the
most cost-effective way to clean up the groundwater by computing time-varying
rates of pumping. Shoemaker's group has also developed an animation, using the
visualization resources (both hardware and personnel) of the CTC, that represents
the effects of different policies and natural chemical and biological processes on
groundwater cleanup. Such animations are crucial for conveying the results of basic
research to the mixed audience involved in setting environmental policy.

-36

157

High Performance Computing Infrastructure and Accomplishments
Sage Grouse-Endangered Species and the US Army

Working with the U. S. Army, Bruce Hannon (Dept. of Geography, UIUC) and Jim
Westervelt, of the U. S. Army Construction Engineering Research Laboratory, have
developed an ecological model for the sage grouse, an endangered species, popula-
tion on an army training base in Washington State. Using the Macintosh software
STELLA, the CM-5 and the GRASS geographical information systems, these re-
searchers were able to optimize the scheduling of trairung exercises to maximize
grouse reproduction and longevity. For each geographic cell, a very large STELLA
model was constructed, representing the grouse at various life stages, different kinds
of plants and predators, soil type and moisture, weather-all the many physical vari-
ables-and also introduce the necessary human activities like tank and troop maneu-
vers on the army range. Each cell, of which there are over a hundred thousand i n
the GIS covering the army base, could contain 100 to 200 variables. This work
demonstrated the efficacy of coupling GIS datasets to ecological models and running
them in a client-server fashion between a Macintosh and the CM-5.

Storm modeling/forecasting

Robert Wilhelmson (Dept. of Atmospheric Sciences, UTUC and NCSA) and his col-
leagues have been able to simulate the development of tornadoes embedded within
larger storms called supercells (producing the largest tornadoes) and along low level
convergence boundaries (e.g. along a thunderstorm cold air boundary) using both
traditional vector supercomputers (NCSA and PSC Cray Research Y-MP and C90 and
NCSA TMC CM-2 and CM-5). Study of these results is leading to a better under-
standing of when tornadoes will develop and to more accurate tornado warnings.
The visualization of the internal dynamics of a severe thunderstorm, created by the
NCSA visualization team in 1989, is perhaps the most widely viewed visualization
of a supercomputer simulation ever made. It had a major worldwide impact on the
adoption of scientific visualization as a working tool of computational science.

Kelvin Droegemeier, a former student of Wilhelmson's, and his colleagues associ-
ated with the Center for Analysis and Prediction of Storms (CAPS), an NSF S & T
Center, have used the NCSA and PSC Cray Research supercomputers to develop the
Advanced Regional Prediction System, a computational model for forecasting se-
vere storms. As of Spring 1994, this model has been used, with data augmented by
the single-Doppler radar network now being deployed by NOAA, in daily weather
reporting on an experimental basis. Because of their use of parallel supercomputers
they have shown that regional storm forecasts based on very high resolution mod-
els are possible with the advent of teraflop computing capabilities in the next few
years. The long-term objective is to improve the prediction of hazardous weather on
scales ranging from a few kilometers (an individual storm) and tens of minutes to
hundreds of kilometers (a squall line or other mesoscale system) and several hours.

Los Angeles Smog

-37-

158

High Performance Computing Infrastructure and Accomplishments

showed that, contrary to EPA policy at the time, it is necessary to control nitrogen
oxide emissions as well as hydrocarbons to control smog. This work formed the sci-
entific underpinning for the Air Quality Management Plan adopted in 1988-89 for
the Los Angeles air basin, the most stringent such plan in the United States. Their
modeling also showed that alternative vehicle fuels, methanol in particular, repre-
sent a worthwhile strategy for improving urban air quality, which influenced inclu-
sion of this policy in the 1990 revisions to the Federal Clean Air Act. This work is be-
ing extended using the combination of the PSC Cray Research C90 and T3D as an
NSF Grand Challenge. The first Forefronts of Large-Scale Computation award, given
in 1989, recognized McRae for this work.

These pioneering computations are leading to practical tools for states to predict air
pollution levels. The Modeling and Meteorology Branch of the California Air Re-
sources Board joined the SDSC Industrial Partners program in 1991. They are run-
ning, on the SDSC Cray Research C90, the Urban Airshed model which estimates
hourly pollutant concentrations. They results are used to estimate maximum pollu-
tant concentrations or population exposure statistics for different emissions con-
trols.

Upper Ocean Mixing

Sidney Leibovich (Dept. of Mechaiucal and Aerospace Engineering, Cornell Univer-
sity) has developed a mathematical model on the CTC IB