Present: Chris Orth (BU Shifter), Ioannis Katsavounidis, Sophia
Kyriazopoulou, Nicola Zaccheo, Massimo Orsini, Valeriano Marrelli,
Mystery PIC
While the run taking went smoothly as usual, there are a few problems
worth noting:
Stop Master -
The spare Stop Master had to be swapped in on SM1 earlier in June.
(See macro-wrong archives at http://budoe.bu.edu/~macro for specifics
dates). The problem hasn't been fully debugged yet, but is probably
due to a flaky connector. The module on SM1 has a minor pathology
that the TOHM face bitpattern doesn'tt latch the center face. The
timeword for the center face is ok, so analyzers are advised (again)
to use the timewords and not the bit patterns in their analysis code.
CAMAC/Attico Tohm problems -
On Wednesday July 14th, a problem with the 6T1-17 TOHM module was
investigated. The symptom was that Erik automatic log files were
recording "fake" TOHM hits from the Stop Master on SM6 (events where
the Stop Master registered a TOP face TOHM trigger, but the TOHM did
not) the T. This problem doesn't persist from run to run. On
Wednesday, Sophia, Ioannis, and CO observed that the TOHM Equipment
(27) readout was missing data was reading all 0s even when Top
triggers were generated manually with the LED. The "fake" Stop Master
events were therefore actual TOHM hits that did not appear in the data
stream. This also means that the WFDs were missing for these hits
(unless another trigger was present in the same box). The CAMAC
module passed all of our available DSP tests, so the CAMAC interface
on the TOHM board is suspected.
On the following Monday night, a problem with the power supply on the
same crate appeared. There was also a problem with the spare so we
ran without mVax 3 until Wednesday.
Next Wednesday we will move this module to the same crate with the
South Face module to see if the reduced current drawn from this crate
improves the behavior. Currently the module seems okay.
Mysterious mVax 1 problem -
On Saturday night (July 18) mVax 1 caused a crash and had to be omitted
from running. We were getting many no Q errors from m1-b2-c3-s22,
m1-b2-c4-s22, and m1-b2-c6-s22 (Stop Masters and interERP). Note that
the errors all come from the same station.
On Sunday, your humble shifter tried cycling power on the SM2 ERP
crate and then reloading LUTS. I was able to run a miniacq for
approximately 30 minutes, at which point I restarted the main ACQ with
mVax 1 included. However after another 40 minutes the problem came
back.
On Monday, Ioannis and Nicola traced the problem to trigger 10 (Slow
Monopole Streamer Tube) events. However, Trigger 10 does not invoke
the readout of any of the affected CAMAC slots. So the problem was
likely with the so called "golden" module (Trigger Pattern module on
the system crate). Massimo removed this module and "cleaned" it by
shaking off some of the accumulated dust and replaced it. Since then
this problem has not recurred.
Problems Related to Power Maintenance -
On Wednesday July 21 we had two planned power outages, for maintenace
on the main PILLER for the MACRO hall. This first outage in the
morning caused the untimely demise of 3 PHRASE power supplies. The
second outage in the evening killed another PHRASE supply. Before
bringing MACRO back online in the evening we had to unplug 3T06-0 from
the TOHM. This channel has a known problem where its thresholds do
not get loaded properly unless loaded manually. This will be fixed on
next Wednesday.
UTC-
A couple of unplanned blackouts in the external labs on this Wednesday
and Thursday caused some glitches with the UTC Master, which in turn
caused problems with the PHRASE synchronization . Eugenio Scapparone
reset the Master on Thursday, and the system is now OK.
Network-
The local network has been a big mess this shift period. There are a
variety of failure modes, and each day at least one at random has been
present. Most predominant is that DECNET from the external labs to
the tunnel is only achievable through axpgs0; the rest of the alpha
cluster fails to connect. This is a problem for automated jobs that
access data in the tunnel directly. Also frequent is the problem
where we cannot connect to the tunnel from the external labs using
TCP/IP or DECNET (or occasionally we can connect, but very slowly),
but telnetting to an intermediate site (budoe.bu.edu, bari.infn.it,
etc) first, and THEN telnetting to the tunnel from there works.
Finally there have been days with no network acess to the tunnel at
all. We hope that when our network administrator returns from sick
leave that these problems will be resolved.
Other Work -
Sophia showed some results from her TOHM crosstalk calibrations last
week, and earlier this year. The crosstalk calibrations are a
variation of the LAMOSSKA LED calibration, where LEDs are fired in one
tank end at a very high level. Here, the LEDs are fired in both
ends. By examining the background rate of the TOHM for unfired boxes
and comparing to the background rate in normal running, Sophia hopes
to measure the level of crosstalk present in the TOHM hardware. This
information is relevant for analyses that make cuts based on the
number of TOHM boxes in an event. Sophia still has a few bugs to work
out of her analysis, but preliminary numbers seem to indicate that the
TOHM background rate is actually lower during these calibrations.
Sophia also presented work from her 500ns width TOHM calibration.
Her rseults showed that some channels still fire even on this narrow
width, but many channels never reached a high enough light level to
measure the TOHM/LI efficiency.
Chris reported that he has finished a second pass reduction on the
Rare DSTS for RDST1-RDS31. The final data set, the "MINI" Rare DST,
fits on a single 10GB DLT tape for this approx. 3 years of running. A
reference copy will be left at Gran Sasso. As reported at the Caltech
meeting, this reduction uses cuts that should be applicable to most
monopole analyses (fast or slow), and focuses on remaining sensitive
nucleon-decay catayzing monopoles. This reduction will be included as
part of the automated run processing.
Chris is also working on LASER calibrations of the WFD and ERP
systems. The first step for this calibration is a study of the
light output as a function of attenuator setting. He attempted to
measure the attenuation using a restricted range of the reference ADC
data where all systems are believed to respond linearly. While
precise fits were obtainable for all but one attenuator wheel, these
results disagree wildly from attenuator to attenuator. This is
unexpected. More calibration runs will be analyzed to confirm that
consistent measurements can be made for each attenuator.
Here's a friendly suggestion for future Shifters concerning Traffic. The
section of autostrade near the airport and the section of the GRA between the
airport and A24 is a complete disaster area, rivaling Boston's "Big Dig". You
can avoid probably 1 hour of traffic by going the other way around the GRA (most
signs say towards "Firenze".) Good luck!
Your humble shifter
Chris Orth