Subject: May 16 meeting minutes -- MACRO data issues
From: Erik Katsavounidis (Erik.Katsavounidis@lngs.infn.it)
Date: Tue May 30 2000 - 10:49:21 EDT
MACRO Colleagues,
As it was advertised, on May 16, 2000 we have had a meeting at Gran Sasso
related to final MACRO data distribution and archiving issues.
The following were present at the meeting: Andrea Donati (LNGS computing
manager), Aurelio Grillo, Alessandra Di Credico, Francesco Ronga, Paolo
Bernardini, Erik and Yiannis Katsavounidis.
The headlines/conclusions/recommendations of the issues discussed can
be summarized as follows:
o The VAX/VMS cluster at Gran Sasso is entering a phasing out status.
It is expected to be completely decommissioned within 3 years from now.
o A VAX/VMS cpu will remain operational at Frascati at least for the next
5 years aimed primarily to serve the neutrino analysis needs of the local
group. A similar long term solution at Gran Sasso does not seem easy
to realize.
o Italian institutions tradionally receiving full MACRO data streams
are lacking on the average 10% of the tapes produced. These are not
expected to be produced, unless each receipient shows explicit interest.
o US institutions with complete data sets are TAMU (Rare DSTs approx. 550GB)
and Caltech (nonWF,nonPHRASE DSTs approx 400GB). BU has received ~50%
of the Rare DSTs- completing this data set is easy to achieve if tapes
and request are submitted while US technical support is present at GS.
o The Gran Sasso VAX/VMS disks will hold in short term (or for as long
as they do not interfere with the cluster decommission) the following
complete classes of data:
- Muon Astronomy DSTs
- Slow Monopole Waveform Analysis DSTs
- RUN-by-RUN diagnostics files
o Long term (6mons - 5 years) data storage at Gran Sasso will be accommodated
through a UNIX server of currently 200GB capacity. Disks available via NFS
to all LNGS cpus through 100Mbps connections will allow the permanent storage
of DSTs like nonWF/nonPHRASE DSTs (400GB) within a year. A first 18GB disk
under this server has started hosting various classes of data for testing
purposes: check out data present on /macro1 under LNGS' unix cpus and/or
NFS_MACRO1:[000000] under LNGS' vms cpus if you ever plan to read MACRO
data off these disks in the near or distant future.
o A MACRO web site with central handling of final papers, plots, software,
diagnostic files etc will be nice to have set up. It is still under
investigation as to where this can be. A general domain www.macro.infn.it
is recommended in order to allow hosting of the www server from any INFN
site and not necessarily through LNGS. 'MACRO Online' a US-maintained site
at Gran Sasso is expected to retire within year 2000.
If you have any particular interest on any of the above issues, please
read through the detailed minutes of the meeting that follow. If you have
any comments, suggestions of preferences please let us know.
1) Andrea presented the current status and planing of the LNGS computing.
- The VAX-VMS cluster will be completely decommissioned within 3 years.
This strictly forbids any spending from LNGS general computing funds
on new VAX/VMS hardware from this point on.
No VMS support of group-owned VMS cpu's both in terms of funds and work
hours of LNGS computing personnel will be granted beyond the point of
the VMS decommission.
The transition will be gradual- some ALPHA cpus will be converted to
digital unix. SCSI goodies (disks,DLT drives) of certain value will
migrate together to unix.
No upgrade of the DLT units on the ALPHA cluster is expected to happen,
i.e., all drives including the TAPE ROBOT will remain at DLT-III (10GB).
A stand-alone MACRO-only DLT-IV is also available. BTW, the DLT ROBOT
is NOT expected to be moved to the unix cluster ever.
- A new unix server (rsgs05) has been acquired. Initially, it offers
a 200GB capacity, most of which can be made available to MACRO needs
in short/long terms. Further capacity can be acquired within 6 months
to a year.
Long-term general maintenance, support and part of hardware (more disks)
acquisition costs will be provided from Laboratory personnel and funds.
This new server is connected through 100Mpbs connections to most of
the LNGS cpus. The unix cluster has now a DLT-IV unit available.
The number of DLT units on UNIX will be increasing according to needs.
It is expected that a *new* robot for the unix cluster will be acquired in
short to medium terms.
2) Francesco described a stand-alone ALPHA cpu running VAX/VMS acquired at
Frascati with intention to remain with its operating system frozen. All
MACRO software and libraries will be frozen there together. Disk capacity
will allow storage of all muon astronomy DSTs plus local users area. The
VAX/VMS package will remain under DEC maintenance for a period of at least
5 years from now while local support and maintenance will be provided by
Francesco et al (Frascati). The Frascati VAX/VMS cpu is primarily to serve
the neutrino analyses needs of the local group.
3) A similar solution at Gran Sasso was also discussed. It appears to be
hard to realize, at least as a long term (5 years) solution. Aurelio raised
two relative points:
- a "frozen" VAX/VMS cpu at Gran Sasso might not be what most of MACRO
analyzers would like for the next 3-5 years. Instead a dedicated digital
unix cpu might be more popular, for example.
- there can be no contribution in the maintenance of a possible VAX/VMS
box at Gran Sasso by local physicists and/or computing personnel.
This issue will be re-examined with the current parameters a few months
before the decommissioning of the last LNGS VAX/VMS cpus (in 2-3 years
from now).
4) The issue of undelivered DLT DATA TAPEs with the full MACRO stream was then
discussed. It is estimated that roughly 10% of the tapes (Aug 95 - now)
were produced only for the LNGS. This applies to Italian institutions only.
There were many reasons that led to such an action, like lack of DLT tapes,
failures of the DLT units, crashes of the cpu's and its disks. Aurelio
will contact Italian groups missing data tapes. It is expected though that
the groups will make lists of the tapes missing and preferrably bring the
DLT-III tapes needed (it has become rather tricky locating DLT-III in
the market lately).
5) We then reviewed the various classes of MACRO data and examined various disk
storage solutions for short (~1year- VAX based) and long term (1-5 years
or more- unix based). In all cases, NFS-availability of the stored data
will allow their processing under all existing at any given moment VAX/VMS
and unix cpus.
The full MACRO data collected since day 1 of the experiment is now 99%
transfered to DLT-III (superseeding ~1500 TA90 and ~300 8mm) and 95%
re-processed in order to produce key-DSTs. It amounts to roughly 400 DLT-III
for a total of 3TBytes. This will never be available on disk; complete
set of archive tapes will remain at Gran Sasso.
A complete set of non-WaveForm, non-PHRASE data is now available in
50 DLT-III for a total of 400GB. This is the COMPLETE RAW MACRO data set
after excluding the waveform and PHRASE equipments. We propose that it
is kept permanently on disk (the rsgs05 solution). A copy of these
50 DLT-III tapes will be kept also at Caltech.
A complete set of the 200MHz WaveForm-based-analyses data will be available
within 15 days in roughly 60 DLT-III's for a total of roughly 550GB.
A copy of these 60 DLT-III tapes will be kept also at TAMU.
All Muon Astronomy DSTs ever produced total roughly 50GB. The ones
after April 1, 1994 are already on the VAX/VMS disks and will be kept
this way indefinetely. We will put on disk also the ones prior to
April 1, 1994 ASAP.
An analysis-filtered 200MHz WaveForm slow-monopole data are available
for the entire running on disk (3GB).
If there is any significant (>5GB) disk requirements not discussed here,
please let us know.
The /macro1 area under LNGS' unix areas is NFS mounted on the VAX/VMS
end too as NFS_MACRO1:[000000] PLease take a few minutes to test-read
various classes of data written there both from the unix and VMS end.
The newest ASTRONOMY MUON DST production is available there under
/macro1/mudst/normal/ from the unix site and NFS_MACRO1:[MUDST.NORMAL]
from the VMS site.
6) Misc:
- We discussed briefly for a MACRO web site that will host the
final (and as much as possible of the past) analysis results,
papers etc. Scanning and archiving of as much of the MACRO MEMOs
as possible could be a nice feature, but it depends on the human
resources at GS that can actually undertake this operation.
`MACRO Online' (running on WSGS02) will soon be off when we'll
unplug WSGS02; we could potentially move it to another MACRO-owned
cpu and archive all desired material there.
It is unclear though who's going to provide long term maintenance
and support. A domain www.macro.infn.it could allow operation of
a MACRO www box from any INFN site; this may end up being easier
to maintain.
- RSGS04.LNGS.INFN.IT users should know that this cpu is now clinically
dead. The external disk mounted therein now mounted on rsgs02 as
/dischovecchio
A. Grillo, A. Di Credico, F. Ronga, P. Bernardini, E.&I. Katsavounidis
This archive was generated by hypermail 2a24 : Tue May 30 2000 - 10:49:28 EDT