From: VAXGS::BRAJESH
Subject: MACRO-USA SHIFT SUMMARY 16th NOV. - 27th NOV.
Date: 27-NOV-1998 16:52:08
Expires: 12-DEC-1998 16:52:08
MACRO-USA Shift Report (Nov 16th to 27th)
----------------------------------------
PEOPLE PRESENT IN THE DETECTOR AREA: BCC, FC, EK, IK, VM, MO, NZ
PEOPLE PRESENT OUTSIDE : SK
PEOPLE PRESENT FOR FEW DAYS : DM, RN
***16-11-97:***
Run 16697 stopped and muVAX5 taken out of DAQ system. Broken fan on a crate.
Run 16698 without muVAX5. Stopped after 75mts to start calibration.
Yes calibration on monday, because we did not get chance to do much
calibration over last several weeks due to HV maintenance.
Run 16699 with muVAX4 and muVAX5 on PISA only.
Calibration runs (16-NOV-1998) (Time taken - approx. 150 minutes):
91210: TOHM X-talk on muVAX1+PISA.
91211: LED/TDC
91212: FMT/TDC
91213: LAMOSSKA
Run 16700: Global run begin with full detector (@12:30hrs).
: 5T05 turned off while run is at @37% (15:07hrs).
: Run stopped for replacement of a splitter box (15:30hrs).
Run 16701: Without muVAX2. 27 minute long run.
Run 16702-16705: Successful Runs - no problems for 24 hours.
5T05 has been put into the DAQ system just before run 16706
started. Two noisy PMT's on 5T05-00,01 were replaced with new
EMI PMT's which arrived only few days ago from Caltech. Voltage
on both PMT's, 1340V. Rate is good. (17-11-98, 15:52hrs.)
Run 16706-16708: Successful Runs. - no problems for 18 hours.
***17-11-97:***
High Trigger (times the average) rates observed on Monopole Tanks
SM1 - none
SM2 - B13(X4), C14(X4)
SM3 - T04(X4), T12(X3)
SM4 - C14(X3), T02(X5), T11(X4), T17(X8)(fixed on 23-11-98)
SM5 - T02(X5), T06(X4)
SM6 - B09(X3), T02(X5)
Looked at the singles rate. Higher rate observed on these tanks.
* - high, ** - higher, *** - very high.
SM1 - T02-0(*), T03-0(*), T10-1(*), T11-0(*), T14-0(***)
SM2 - B13-0(***), C01-0(*), C06-0(*), C14-0(***), E07-1(***), T14-0(***)
SM3 - T06-0(**)/1(***), T07-1(***), T11-(*), T12-0(***), T13-0(***)
SM4 - B05-1(**), B14-0(***), C12-0(***), W01-1(**), T02-0(***), T03-1(**),
T04-0(**), T07-1(**), T10-0(**), T11-1(***), T12-1(***), T13-0(**),
T15-0(***), T16-1(**), T17-0(***)
SM5 - B11-1(***), T01-0(**), T02-0(**)/1(**), T03-0(*)/1(*), T04-0(**),
T06-0(***), T07-1(***), T08-0(***), T10-1(**), T11-0(**), T15-0(**),
T16-0(***)
SM6 - B09-0(**), B13-0(**), B14-1(**), T02-0(***)/1(*), T07-0(***),
T08-0(**), T14-0(***)
***18-11-98:***
Calibration day.
Run 16709: PISA only. SM 3/4/5/6.
Run 16710: PISA only. SM 1/2/5/6.
Run 91214: LASER Lower - SM 1/2.
Run 91215: LASER Attico - SM 1/2. - Problem with laser system. Data not good
for calibration but good for debugging.
The problem may be with attenuator. We
do not have any spare attenuator.
Run 91217: LED/FMT - SM 3/4.
Run 91218: LED/TDC - SM 3/4.
Run 91219: LASER Lower - SM 3/4.
VSMAC1 crashed. Rebooted - but does not help.
Sandro MARINI from Frascati looking into it.
***NO BACKUP DAQ SYSTEM SYSTEM AVAILABLE.***
***19-11-98:***
Run 16711: Finished gracefully.
About 11:00pm when 16711 finished and 16712 started in auto mode the system
crashed. It was found stopped at 11:30pm. When Paulo Lipari tried to start
the DAQ system, he had several problems. Finally all muVAXes were rebooted
and DAQ started nicely.
Run 16712 : ******Does not exist.******
Run 16713 - 16719: Finished without any problems.
4T17 was taken out of DAQ system after RUN 16714. BCC/NZ looked at both
both PMT's on 4T17-0. Both have very have high rates. 4T17-00 has very high
rate and 4T17-01 has also high rate. NZ/VM looked at the PMT's after
draining the oil. Several mistakes found in putting the system together.
They have put back the system as per the check list and HV is on but
no oil is there. Trying to understand if the problem are with PMT's or with
putting the system together.
The high rate on PMT's on SM 4/5 on side zero may be due to many small
factors but the most severe is due to Helium leak. Very deiled studies are
going on. ***SEEMS LIKE HELIUM LEAK IS BECOMING A PROBLEM.***
SK/IOK is going to provide more detailed information.
***20-11-98:***
Run 16720: Run ends after 4.8hrs at 22:40pm. Abnormal end of run.
Major power failure on 19th night due to snow storm. Run16720 ended
abnormally. All electronics off on MACRO roof. Italian shifter(MM) alerted.
Arrives in tunnel. All crates switched off in order to prevent any damage
at restart. Also switched off VXMACA/VXMACB, all the workstations etc.
Switched off PC on GAS system.
***21-11-98:***
Power back at 11:50 am. High temperature in computer room. Alarm on GAS
system. Failure in reboot on computers. Fluctuations in power supply.
Another temporary Blackout.
Gas system restarts at 13:15hrs.
Problem with switching on VXMACA/VXMXCB.
VXMACB does not reboot properly. HV program resides on it.
Finally HV on PMT's turned on manually.
Several fan units and power suppllies switched off. All of them turned on.
When almost everything was ready by 19:30 hrs another power failure happens.
Power is back at 22:00 hrs.
***22-11-98:***
VXMACB still dead.
Another attempt to restart the run with all 6 muVAXs on VXMACA.
Link Problem.
Finally RUN 16722 started at 1:00 am with all 6 SM for PHRASE only.
Already 25 hrs 20 minutes lost.
Run 16722 goes on for almost 12 hrs.
We do not see any signal on monitor for PISA on SM1 ans SM2. Run 16722
ended.
Run 16723 started with SM 3,4,5and6, PISA only. Run 16723 ended at 16:27hrs.
Run 16724 started with all 6 SM for PISA only. Run goes on smoothly.
Run 16724 crashes at 4:15am.
***23-11-98:***
Fan unit on SM4 in PISA found broken. LEC_HV off. Some ST's off.
TECHNICAL TEAM (MO+NZ+VM+Jan Reynolds) at work. They have fixed the broken
fan on PISA unit. They also found that TOHM power supply on SM1 had problem.
This power supply was damaged when second power outage happened on saturday
night. That's why IOK and MM, who were in tunnel late in the night had
problem starting the run on all six muVAX's. Finally most of the problem
with electronics and DAQ system fixed.
Global Run 16725 started at 10:45 hrs.
Thanks to IOK/MM/MO/NZ/VM/JR for making the things work.
DGN does not work. Made to work.
The support from LAB in time of crisis is almost non-existent.
We lost nearly
***60 HOURS *** of GLOBAL DATA and
***33 HOURS *** of PHRASE DATA.
***PULSER CALIBRATION B1 vs B2 TO BE REDONE AGAIN.***
VXMACB is still dead. Experts (at Frascati) informed on saturday itself.
Seems like VXMACB was not brought down properly.
Now VXMACB thinks itself as VXMACA.
""EVEN COMPUTERS HAVE WRONG NOTIONS ABOUT THEMSELVES.
AFTER ALL MORTAL HUMAN BEINGS INVENTED IT.
THIS IS HOW VIRUS TRANSFERS FROM HUMANS TO COMPUTERS.""
VSMAC1 IS STILL UNOPERABLE AS BACKUP DAQ SYSTEM.
MARINI from Frascati trying to see what is wrong with VSMAC1.
Once VSMAC1 starts operating as backup DAQ system (hope so), we will go to
the backup system and bring down VXMACA/VXMACB and try to see that we can
fix the "brain" of VXMACB. If that works then "life is again rosy and
boring" otherwise we any how need to get VSMAC1 fixed as backup system so
that we can ask DIGITAL to do **brian surgery** on VXMACB.
More stories to follow.
VSMAC1 is hopefully fixed. According to Sandro MARINI the
BUFFER MANAGER software was badly installed that's why it crashed on last
wednesday (18-11-98). More tests to follow.
RUN 16725 ended at 15:25 hrs.
BCC/IOK tried to start run on VSMAC1, the backup system, so that they can
work on VXMACB. The run did not begin on VSMAC1. Problems. IOK called
SANDRO MARINI and talked. Finally VSMAC1 WORKS.
RUN 91224 (with all six muVAX's) started after 20minutes at 15:45 hrs.
***This is a GLOBAL RUN although RUN NUMBER is like a CALIBRATION RUN.***
16:00 hrs: BCC/IOK/MM tried to reboot VXMACA and VXMACB. VXMACB still thinks
it is VXMACA. VIRUS of self-indulgence needs special tretment to cure the
disease. IOK called Sandro MARINI, who will call the real doctors at
Digital to cure VXMACB. We decided to continue running on VSMAC1.
RUN 91224 going on smoothly.
VSMAC1 has 5 Million free blocks. This should give enough disk space to last
for 6 runs that is for next 36 hours. We will keep running on VSMAC1 until
the folks from DIGITAL fix the problem on VXMACB and then we will switch to
usual system. Copying of file and keeping enough free space will be taken
care of.
MO/NZ replaced both the PMT's on 4T17-0 with new EMI PMT's which came from
Caltech few days ago. The voltage applied are 1210 and 1220 on 4T-00,01
respectively. 4T17 will be put into the DAQ system at appropriate time.
Next we will work on 5T02-0. PMT's from 5T05 should be put into dark box and
looked at after couple of days.
***24-11-98:***
Run 91224 to 91228 finished normally. Channel 4T17 put back in the DAQ
system before run 91228 started. Channel 5T02 taken out of DAQ system
before run 91228 started.
Run 91229 started at 17:16.Run ended at 18:57 to move from VSMAC1 to VSMACA.
MOVED from VSMAC1 to VXMACA for Global run because VSMAC1 does not have
enough disk space to keep running for whole night.
Now the GLOBAL runs will have run numbers in 16*** series as of normal run.
RUN 16726 started.
*****MORE ON DIGITAL SAGA*****
DIGITAL was informed of the VXMACB problem yesterday. No response till 5pm
today. Someone called "Bocci" from DIGITAL Torino called. Gave hard time to
Italian shifter that we must have screwed up the computer and they are not
responsible. Bla, bla, bla. I wish I knew Italian. Finally MARINI from
Frascati called DIGITAL. He had to threaten to cancel the contract worth
45M Lira. Finally the DIGITAL guys came to their senses. Hopefully now
someone will come. We will know the details tomorrow.
***25-11-98:***
RUN 16726-30 went nicely.
5T02 was put back in acquisition just before Run 15729. No PMT's replaced
on 5T02 because we can't replace PMT's at this speed. We do not have enough
of them.
DIGITAL technician will come on friday. We reported the problem to digital
on monday itself.
***26-11-98:***
Guards called Italian shifter at 00:30hrs to tell that a PISA crate was in
alarm. Alarm found on SM4, Rack9, Crate4. Last trigger from muVAX4 at 00:12.
Run 16731 ended (71 mts long run) because of error on muVAX4. muVAX4
removed from DAQ system. Run 16732 & 16733 are junk runs. Total 45 minutes
lost in the process.
RUN 16734 with muVAX 1, 2 and 3 only. *** NO PHRASE RUN GOING ON. ***
RUN 16735 with muVAX 1, 2 and 3 only. 2 hour long run. Run ended at
10:20 am to include PHRASE system. Run 16736 is a junk run.
RUN 16737 with muVAX 1,2,3,5,6 only. SM 3 and 4 not in PHRASE.
WE lost 9 hours 20 minutes of PHRASE data.
MO/NZ/VM tried to fix the crate. Crate included in the system.
Run 16737 ended after 5 hrs of running.
Run 16738 started with all six muVAX's.
Run 16738-16740 finished normally
***27-11-98:***
Run 16740 finished at 8:26 a.m. While starting next run the auto system had
problem because of not adequate disk space on VXMACA. Run stopped at 8:27am.
Moved to backup system. Had to do reboot. 75 minutes lost.
Run 91230 (Global run) started on backup system. Finished gracefully.
Run 91231 going on.
DIGITAL technician working on VXMACB. Several problems.
HOPEFULLY IT WILL BE FIXED TODAY.
****************************************************************************
UPDATE on RDST production from Ioannis KATSAVOUNIDIS
As of Nov. 19, 1998 the following 31 RDSTs have been produced in
both RAW and ZEB format. 21 of these have been copied and sent to the
US (RDST6 - RDS25). The other 10 (RDST1-5 and RDS26-30) will be copied
as soon as DLTs become available at Gran Sasso. It takes roughly 2
hours for a tape-to-tape copy, so these 10 rdsts can be copied for
CIT,BU and TAMU in a week.
RDST1 : RUN 12827 (26-SEP-1996) - RUN 13033 (09-NOV-1996)
RDST2 : RUN 13034 (09-NOV-1996) - RUN 13219 (20-DEC-1996)
RDST3 : RUN 13220 (20-DEC-1996) - RUN 13406 (24-JAN-1997)
RDST4 : RUN 13407 (24-JAN-1997) - RUN 13572 (27-FEB-1997)
RDST5 : RUN 13573 (27-FEB-1997) - RUN 13759 (03-APR-1997)
RDST6 : RUN 13760 (03-APR-1997) - RUN 13893 (29-APR-1997)
RDST7 : RUN 13894 (29-APR-1997) - RUN 14059 (31-MAY-1997)
RDST8 : RUN 14060 (31-MAY-1997) - RUN 14226 (28-JUN-1997)
RDST9 : RUN 14227 (28-JUN-1997) - RUN 14349 (22-JUL-1997)
RDS10 : RUN 14350 (22-JUL-1997) - RUN 14505 (23-AUG-1997)
RDS11 : RUN 14506 (23-AUG-1997) - RUN 14624 (18-SEP-1997)
RDS12 : RUN 14625 (18-SEP-1997) - RUN 14773 (16-OCT-1997)
RDS13 : RUN 14779 (16-OCT-1997) - RUN 14898 (11-NOV-1997)
RDS14 : RUN 14899 (11-NOV-1997) - RUN 15048 (10-DEC-1997)
RDS15 : RUN 15049 (10-DEC-1997) - RUN 15197 (07-JAN-1998)
RDS16 : RUN 15198 (08-JAN-1998) - RUN 15349 (05-FEB-1998)
RDS17A: RUN 15288 (25-JAN-1998) - RUN 15394 (14-FEB-1998) -> NEW CUTS
RDS17B: RUN 15395 (14-FEB-1998) - RUN 15485 (05-MAR-1998) -> NEW CUTS
RDS18 : RUN 15486 (05-MAR-1998) - RUN 15579 (24-MAR-1998) -> NEW CUTS
RDS19 : RUN 15578 (24-MAR-1998) - RUN 15667 (13-ARP-1998) -> NEW CUTS
RDS20 : RUN 15668 (13-APR-1998) - RUN 15764 (04-MAY-1998) -> NEW CUTS
RDS21 : RUN 15765 (04-MAY-1998) - RUN 15853 (23-MAY-1998) -> NEW CUTS
RDS22 : RUN 15854 (23-MAY-1998) - RUN 15960 (13-JUN-1998) -> NEW CUTS
RDS23 : RUN 15961 (13-JUN-1998) - RUN 16062 (04-JUL-1998) -> NEW CUTS + 15896
RDS24 : RUN 16063 (04-JUL-1998) - RUN 16156 (23-JUL-1998) -> NEW CUTS
RDS25 : RUN 16157 (23-JUL-1998) - RUN 16239 (10-AUG-1998) -> NEW CUTS
RDS26 : RUN 16240 (11-AUG-1998) - RUN 16329 (30-AUG-1998) -> NEW CUTS
RDS27 : RUN 16330 (30-AUG-1998) - RUN 16421 (17-SEP-1998) -> NEW CUTS
RDS28 : RUN 16422 (18-SEP-1998) - RUN 16514 (08-OCT-1998) -> NEW CUTS
RDS29 : RUN 16515 (08-OCT-1998) - RUN 16598 (27-OCT-1998) -> NEW CUTS
RDS30 : RUN 16599 (27-OCT-1998) - RUN 16704 (17-NOV-1998) -> NEW CUTS
****************************************************************************
I just returned my personal magnetic batch and the magnetic batch for my
car. I am free from being another binary combination in the technological
world. I am free at last.
Respectfully Submitted.
Brajesh C. Choudhary
CALTECH(MACRO-USA)
27-NOV-1998 (5p.m)