Present for a significant part of the last two weeks: B. Choudhary, S. Coutu*,
Y. Huang, E. Katsavounidis, I. Katsavounidis, H. Kim, S. Kyriazopoulou,
M. Orsini, P. Saggese, S. Stalio
Flickering in and out of existence: L. Gray, D. Michael, B. Nolty,
K. Scholberg, C. Walter
* Your humble reporter
While most everybody was gallivanting in that town with the crooked tower,
having a piece 'a Pisa pizza (no, really, say it out loud), some of us were
busy holding down the fort. Here's some of what happened, according to random
notes I took during the not-so-weekly meeting. If you did something important
that I forgot to mention, tough. Life isn't fair.
DATA COLLECTION
- Data-taking was fairly uneventful for the first week. Of course. Everybody
was gone. Exciting highlights included repeated trips of the same few
streamer tube CAEN HV channels (where the HV at which they were reset was
adjusted by trial, error, and playing chicken). After any given trip, the
dead time for the particular uVAX affected went up from the normal rate
(these days) of ~2% to in excess of ~7-8%. There were a few instances of
overnight trips that were only corrected the following morning. On Oct 23,
one hot TOHM channel was quickly traced to a newly defunct PMT (3T07-00),
which was disconnected before it could cause too much trouble (and hence for
a few days 3T07-0 was operated with only one PMT). A few aftershocks of the
Assisi earthquake have rumbled through the area (I would estimate a
magnitude 3.5 at L'Aquila, though this was measured through a haze of
Centerba); no noticeable consequences followed (except a bad headache).
- The second week, all of Hades broke lose, and the feces hit the ventilator
in a big way. The problems began during the night of October 28-29,
presumably related to the torrential 0.001 inches of snow which blanketed
the area. In the morning of the 29th (a Wednesday), it was discovered that:
o the run had crashed at about 1 am;
o prior to that, TOHM activity was high in most 5T channels, with a very
high TOHM trigger rate in 5T10;
o an ethernet cable connecting the Pisa uVAX for SM 5,6 to the rest of the
acquisition system had suddenly defuncted;
o a QTP crate in SM 3,4 had a dead fan, somehow causing the entire crate
not to Q-respond to the acquisition system, and somehow preventing
uVAX 2 from rebooting properly.
Calibrations were performed (more on this below), some maintenance and
repairs were done (new ethernet cable for that Pisa uVAX, new fan for that
crate, no obvious fix for the TOHM problem as it seemed to go away all by
itself). The following morning (30th), we found that the run had crashed
again during the night, with the same QTP crate on the lamb. This time,
a loose connector was retightened on the crate's power supply, which seemed
to finally do the trick. The following morning (31st), an alarm on the
overall MACRO UPS system was interpreted by the lab's engineers in charge
of it as meaning that the batteries were recharging following some ENEA
power problems. Nope, the problem, as it was only realized later during
the day, was that the batteries for the unit are actually on their last
leg, and are therefore being automatically bypassed, so that MACRO is being
fed unfiltered, ENEA power directly. Scary? You bet! The multiple failures of
the last couple of days are very likely related to this. This afternoon,
repeated fluctuations in the power in the underground laboratories (no
doubt due to some persistent fog outside) caused various LeCroy HV supplies
to trip. The danger of damage to the supplies prompted a decision to
shut down the HV to all scintillator systems in SM 3-6, keeping only SM 1,2
alive for now (the LeCroy units for them have proven a bit less susceptible
to power fluctuations). Until such time as replacement batteries for the
UPS unit are provided (unlikely to happen in the next several days,
especially with tomorrow being a big Italian Holiday), and for as long
as ENEA power remains so unstable, the scintillator systems of MACRO will
have only a bare minimum kept operating for gravitational collapse
sensitivity. Streamer tubes are left on, as they seem not to suffer unduly.
This undead status for MACRO seems particularly fitting, on this
Hollowe'en's Eve... One more note to the strangeness of it all: all non-Pisa
events currently being generated have event number 0. It is currently
being speculated that some damage may have occured to the synchronization
modules (power cycling of their crates didn't help).
CALIBRATIONS AND MAINTENANCE
- A full set of calibrations (including LAMOSSKA) were performed on SM 5,6
(Oct. 22) and SM 1,2 (Oct. 29). As part of these procedures, an intensive
debugging effort revealed a number of problems with the laser calibrations
(swapped cables, a reference PMT with a very low gain that introduced
time-walk problems, an ADC gate that was open for much too long, etc...).
A careful comparison of the relative amplitude of signals at both tank
ends during laser calibrations revealed yet another bad HV splitter box, with
an unsoldered wire inside (1C14-0, about the 12th found in this
manner within about the last year). This was repaired.
- Other repairs included: the replacement of a 5V regulator that made the
SM2 LIP trigger work again, the replacement of that dead PMT 3T07-00 (Hwi's
introduction to the joys and pleasures of MACRO oil wrestling), the
refurbishment of a borrowed Philips discriminator module (needed temporarily
for the CSPAM monopole trigger in uVAX 3 while a replacement is on order;
this borrowed module had been modified from its original condition, yielding
a higher-than-normal trigger rate since installed about a week before), and
the swap of a broken LED switchbox for SM1 (1T01-08, side 1).
OTHER STUFF
- Work is progressing on making the wrong list Web-friendly; check out:
http://wsgs02.lngs.infn.it:8000/macro/wrong ,
a wrong list home page,
http://wsgs02.lngs.infn.it:8000/macro/wrong/list.html ,
a compact list of entries, with links to:
http://wsgs02.lngs.infn.it:8000/macro/wrong/db.html ,
the complete wrong texts (*not* the completely wrong texts).
- The FARFALLA package has been installed at Gran Sasso, and work is in
progress to use it to produce DST's for LIP, ERP muon and LAMOSSKA events.
Monopole DST's could be produced in such a way, if desired.
- The three new "US" collaborators from Caltech (Hwi, Brajesh and Yinzhi) were
initiated to the deep mysteries of the MACRO shift and acquisition
procedures, including such perennial classics as the solution to the
thorny "unable to map the global section" problem, the ever-popular
"No-Q-Response" blues, and the show-stopping, esoteric blanking out of
the online log screen. In addition, Hwi has been admitted to the rank of
Private Shift Worker 2nd Class, a ceremony which involved baptism by oil.
Your shifty character,
Stephane