On May 20, Run 14002 stopped at ~8pm due to a Pisa problem in SM1/2 (system
crate). The Italian shiftworker contacted PS and was instructed to re-start
the run at 9:00 pm without the Pisa microvax. Unfortunately, autorun was
not set to on and the run stopped at 2:30 am on Wednesday (May 21) morning
and no data was collected after that (until next morning).
On May 21, while addressing the SM1/2 Pisa problem, another problem
with a Pisa power suplly in SM4 showed up that kept Pisa in SM3/4
out until ~5pm that day.
On Wednesday morning
(maintenance day) the SM4 fanouts and WFDs were unplugged to add the fix
to them but the wrong capacitors were shipped from BU and new capacitors
had to be obtained. While playing with the fanouts a chronic problem
with them was discovered: When the lemo cables are wiggled, large amounts
of noise appear on the outputs. It was discovered that many of the
grounding lugs on these were loose, resulting in a poor ground (more
on this later - believe it or not it gets worse). After tightening the
lemos on the SM4 fanouts (resulting in much better looking signals)
the detector was put back in aquisition at 6:30 pm. MACRO ran smoothly
until the middle of the next run on Thursday morning at 3:30 am when
a TOHM in SM5 decided to freak out followed by a SM2 TOHM a few minutes
later (this turned out to be caused by a bad power supply connector).
When the next run started, the same SM5 TOHM as before acted up but then
settled down. At 5:00 am the SM4 Pisa crate power supply failed and
crashed the run.
Although this amazing number of problems may at first appear unusual it
should be remembered that MACRO was welcoming the new run coordinator and
it was the first week of this reporter's shift. As appealing as it is to
place the blame on these factors, it should be realized that the two things
that caused the most loss of data were 1) the dramatic increase in the
failure of a particular Pisa crate (which is actually now fixed) and
2) The problem with the default setting for autorun.
The aquisition system should be fixed so that autorun=on is automatically
appended. To lose six hours of data for this kind of thing is stupid.
In the second week (5/21 - 5/30) there have been no crashes and the run has
proceeded rather smoothly but with slightly higher rates.
By far the most amount of time these last two weeks has been spent fixing
the problems associated with the Boston Fanouts (FO) and Waveform Digitizers
(WFDs). To make this system sensitive to monopoles
(a rather urgent issue considering that we are two years into a five year run)
both the FO channels going to the WFDs and the WFDs themselves needed to be
modified. On the FOs the coupling capacitors in the WFD outputs had to be
replaced to increase the AC coupling, thereby reducing the positive (and
negative) undershoot. On the WFDs the capacitors between the input amplifier
and the discriminators had to be reduced to act as a high pass filter thus
sharpening the undershoot. In addition, eight capacitors which in the past
have been known to cause oscillations in the pedestals were removed since
they were accessible during the fix (this was done systematically in
SM5/6/1/2 but not in SM3/4). As I mentioned previously, a known problem
was noted again as soon as we started testing the fanouts. The lemo inputs
on the fanouts are isolated from the front panel by means of two plastic
shoulder washers (to eliminate ground loops). Unfortunately, the preferred
way of mounting lemo connectors (on the board rather than the front panel)
was not used. It turns out that
a good fraction of the ground lugs on these lemos were loose and this is
what caused the noise when the connectors were wiggled. Note that some noisy
channels in the past have been fixed by just wiggling the cables. It is
almost certain that all of the nuts on these lemo connectors had been
tightened when they were first installed.
We now believe that the grounding lugs got loose when the
braid was soldered on, heating and softening the plastic shoulder washers.
The reason we believe this is that when we tried to solve the problem by
first tightening the nuts and then soldering the braid and lug directly to
the nut (which turned out to be a big mistake) they loosened considerably.
This required a great deal of painful desoldering and disassembly.
Ultimately it was decided to just make sure the nuts were tight, or
alternatively on some modules, to additionally solder the braid to the
back part of the lemo so that it could not not lose its ground. Fixing the
grounding problems on ALL the lemo outputs of the FOs became part of the
fix that consumed nearly all of the second week (5/27-5/30).
In an all out effort that would make Henry Ford proud, EK, IK, MO, PS,
SS and GT set up an assembly line to do the FO and WFD fix in only four days.
In addition to the fixes, the work involves removing all the cabling, removing
all the modules, cleaning them and the crates, and replacing bad fans.
This was done without stopping the run for the unaffected microvaxes. In the
meantime LG performed the LED runs for the WFD tests by using the down time
for the modifications to do these checks. The schedule for the fixes was as
follows: (Wed. 5/21 SM4 FO (with great difficulty)),(Tues. 5/27 SM4,5,6 WFD),
(Wed. 5/28 SM5 FO), (Th. 5/29 SM6 FO, SM1,2 WFD), (Fri. 5/30 SM1,2 FO).
IT IS STRONGLY ENCOURAGED THAT PERSONS WISHING TO DO ANALYSIS USING THE WFDs
START DOING SO NOW to assess their performance after the fix.
IK reports that the work updating the calibration
constants is proceeding faster than expected at a rate of 1.5 weeks for
each day spent. The official data base is now good until late Sept.1986
and the work database is good until mid Jan 1997. At this rate it will take
about one more week to catch up to the present.
MO is trying to improve the LAMoSsKa circuit by using an amplifier
with a faster slew rate. SS has been making plots of ERP energy reconstruction
for LaMoSsKa and non-LaMoSsKa events.He sees a tail on the non-LaMoSsKa events
for SM1 but not SM2. It is likely that one badly reconstructed box in SM1
with lambda=300 m is the culprit! EK in an e-mail to the rare particle
working group and other interested parties has pointed out that the output
of LaMoSsKa will be made into a trigger, ie to read out the waveforms.
To do this we will need the timing of LaMoSsKa relative to the rest
of the triggers. A standalone module (perhaps the old latching scaler used
in the past for monopoles) will do the timing. In the present scheme, LaMoSsKa
info is latched in a stand-alone bit register which is read by Equipment 36.
With its upgrade to a trigger (#11?), LaMoSsKa will involve reading
a newly created equipment (#37?) that will take care of the time words,
equip 40 (the standard Stop Master readout) and possible SM-wide WFD readout.
LG has done big charge calibration LED runs for the WFD tests during
the fix installations. On Wed, Th. and Fri she did these tests for SM4, SM5
and SM2 respectively AFTER the modifications were made. This was done to
determine the thresholds for the LaMoSsKa stop and to make sure the WFDs
were performing properly. All is well except for one WFD in SM4 which appears
to have a problem.
On a lighter note: The crew here was discussing what will happen in a few
years to MACRO. Various options were mentioned including the possiblity that
it might be dismantled. We all thought that this would be a waste considering
that the Italian government is spending huge amounts of Lire to finish off
LVD and keep it operating as an "observatory." The suggestion was made to
sell MACRO to LVD for say $20M! That way they could REALLY claim to have the
"Most Powerful Scintillation Detector in the World!" For Lire on the Cento-Lire
they could finally do some science and the MACRO collaborators could receive
valuable funds to spend on their next incredibly wonderful experiments. For
nostalgic reasons the LVD folk could even fill the attico with some of their
boxes. For a price, we could even provide the LVD folk with a "service
contract" provided by four of the world's finest technicians! Just an idle
thought that crossed our minds while installing the 6000th capacitor in the
WFDs and FOs.