GSI Forum
GSI Helmholtzzentrum für Schwerionenforschung

Home » PANDA » PandaRoot » Bugs, Fixes, Releases » TPC digitization blocks everything for too many cluster events
icon8.gif  TPC digitization blocks everything for too many cluster events [message #10769] Thu, 03 June 2010 22:04 Go to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *54-82-r.retail.telecomitalia.it
Dear all,
I have noticed, producing many events with different momentum values, that sometimes the digi macro persists on some well defined events even for tenth of minutes!
In particular, the detector which is "blocking" the digitization is the TPC.

This is a sample of "blocking" event, produced when I was trying to simulate one single pion events from 0.5 to 4 GeV/c:

PndTpcClusterizer:: 20278 clusters created
41409 electrons arriving at readout
Aggregating drifted electrons into avalanches  finished.
41409 Avalanches created
0 aggregations done.
77354 Signals created
PndTpcElectronicsTask::Exec
Building up padmap ...finished. 317 pads hit
........  


After 20 minutes it is still blocked at the same event. This happens also many times... in this particular case 20278 clusters created, 41409 avalanches.

I copy a "normal" TPC event:

PndTpcClusterizer:: 674 clusters created
1214 electrons arriving at readout
Aggregating drifted electrons into avalanches  finished.
1214 Avalanches created
0 aggregations done.
2260 Signals created
PndTpcElectronicsTask::Exec
Building up padmap ...finished. 287 pads hit
...........
166 Digis created


In this case "only" 674 clusters. Digitization time... less than 1 second.

I can understand that for large events the required digitization time is high, but 20 minutes are really too much, I think. I am not sure if, at the end, we will be able to analyse them or they will be simply too much noisy. In the latter case, maybe a rejection of events with a too high number of clusters/avalanche in tpc could help making the digitization a bit faster, without losing too much signal. Or maybe sometimes the code enters inside a too alrge loop that could be optimized, I don't now.

I would call for comments from the TPC experts.
Re: TPC digitization blocks everything for too many cluster events [message #10775 is a reply to message #10769] Sun, 06 June 2010 20:21 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *117-80-r.retail.telecomitalia.it
Hello,
this weekend I have run many events, and the problem appeared in a ... strong way!

At the event 5533:

PndTpcClusterizer:: 30582 clusters created
65536 electrons arriving at readout
Aggregating drifted electrons into avalanches  finished.
65536 Avalanches created
0 aggregations done.
126730 Signals created
PndTpcElectronicsTask::Exec
Building up padmap ...finished. 232 pads hit
........
1575 Digis created  


It took 24 hours, from saturday 20:04 to sunday 20:13 to digitize one single event.

I think, as soon as possible this should be fixed, if not it results hard to simulate "many" events. I have digged a bit in the code, and if I have understood well PndTpcElectronicTask should be the guilty guy. But I have not understood how.
Re: TPC digitization blocks everything for too many cluster events [message #10776 is a reply to message #10775] Mon, 07 June 2010 00:02 Go to previous messageGo to next message
Felix Boehmer is currently offline  Felix Boehmer
Messages: 149
Registered: May 2007
Location: Munich
first-grade participant

From: 62.196.29*
Dear Stefano,

as I told you, I am unfortunately unable to investigate this in detail right now, since I have limited internet access in my hotel.

Can you give me any more information on the type of event? I would guess some low momentum decay product is spiraling in the chamber, producing tons of digits.

Please supply me with more information. Until then I suggest you introduce a cutoff on the avalanches.

Cheers

Felix
Re: TPC digitization blocks everything for too many cluster events [message #10778 is a reply to message #10776] Mon, 07 June 2010 16:05 Go to previous messageGo to next message
Malgorzata Gumberidze is currently offline  Malgorzata Gumberidze
Messages: 98
Registered: June 2005
continuous participant
From: *in2p3.fr
I have the same problem as Stefano.

I'm using pandaroot revision 8473 for my simulation.
What i do is to simulate with pgun electrons with the
momenta 0-5GeV/c. I'm running 10k events. Simulations get block
at the event 1201, and it take a lot of time, till it pass
problematic event. I'm already waiting 20 minutes.

this is a last print out which i get:

***************** PndEmcMakeBump, event: 1201 **************
Digi at (66, 68) was a local max. Energy = 1.46065
EMC header: fired crystals= 26, digi= 15, Total energy= 1.66663 [GeV],
Reconstructed clusters= 3, Total energy in clusters= 1.65603 [GeV]
-I- PndTofHitProducerIdeal: 2 TofPoints, 2 Hits created.
-I- PndTofHitProducerIdeal: 0 SciFTofPoints, 0 sciF Hits created.
PndTpcClusterizer:: 26737 clusters created
51323 electrons arriving at readout
Aggregating drifted electrons into avalanches finished.
51323 Avalanches created
0 aggregations done.
103810 Signals created
PndTpcElectronicsTask::Exec
Building up padmap ...finished. 397 pads hit

Few months ago i was doing the same kind of simulations with the older version of the revision, and i didn't have this kind of problem.

gosia

Re: TPC digitization blocks everything for too many cluster events [message #10782 is a reply to message #10769] Tue, 08 June 2010 08:49 Go to previous messageGo to next message
Jens Sören Lange is currently offline  Jens Sören Lange
Messages: 193
Registered: June 2005
first-grade participant
From: *physik.uni-giessen.de
Hi all,
yes, the same happens not only for pgun, but sometimes for EvtGen generated events, which are "normal" input events (i.e. there is a priori no low momentum particle in the generated event). It means: if these are spiraling tacks, they must originate from Geant.
However, it is not blocked. It is true that it takes a lot of time, but it continues. I had events which needed ~2 hours (same as Gosia, ~30000 clusters or more), but they are always finished. One can also see in the CPU comsumption ("top") that it continues to run.
cheers, Soeren
Re: TPC digitization blocks everything for too many cluster events [message #10783 is a reply to message #10782] Tue, 08 June 2010 09:10 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
I agree, it is not blocked.
In my case, the guilty event needed 24 hours, but after the digitization continued. The only problem consists in how many "guilty" events are inside one sim file.
Re: TPC digitization blocks everything for too many cluster events [message #10786 is a reply to message #10783] Wed, 09 June 2010 08:03 Go to previous messageGo to next message
Sebastian Neubert is currently offline  Sebastian Neubert
Messages: 282
Registered: March 2006
Location: Munich
first-grade participant

From: *natpool.mwn.de
Dear colleagues,

there have been a few modifications in the ElectronicsTask (or more correctly in the components it uses) that have been introduced while we were studying the real data from our test-chamber. If the problem has appeared after these modifications then one should look into the PulseShapeAnalysis or the ClusterFinder.

Obviously there are several options to circumvent the problem but I would strongly suggest to cut out "bad" events (with a lot of primary clusters from a spiraling particle) since in fact they will be reconstructable most of the time. So this is only an option if you need a fast simulation and do not plan to study any efficiencies!

I will look into the digitization if I find a bug there but it might take some time. In the meantime it would be helpful if you could turn on all digitization output for the tpc (SetPersistence for all tasks, plus SetSamplePersistence for the ElectronicsTask) and plot the Signal amplitudes, Sample amplitudes and Digi amplitudes (A few events are enough) Maybe we can tune some cuts there.

Cheers! Sebastian.


Sebastian Neubert
Technische Universität München
Department Physik E18
sneubert@e18.physik.tu-muenchen.de
tel: +49-8928912592

[Updated on: Wed, 09 June 2010 08:03]

Report message to a moderator

Re: TPC digitization blocks everything for too many cluster events [message #10787 is a reply to message #10786] Wed, 09 June 2010 10:05 Go to previous messageGo to next message
Sebastian Neubert is currently offline  Sebastian Neubert
Messages: 282
Registered: March 2006
Location: Munich
first-grade participant

From: *natpool.mwn.de
Hi!

I spotted a mistake in my recent post - what I meant was: "I would strongly suggest NOT to cut out "bad" events..."

Sebastian.


Sebastian Neubert
Technische Universität München
Department Physik E18
sneubert@e18.physik.tu-muenchen.de
tel: +49-8928912592
Re: TPC digitization blocks everything for too many cluster events [message #10789 is a reply to message #10769] Wed, 09 June 2010 10:29 Go to previous messageGo to next message
Jens Sören Lange is currently offline  Jens Sören Lange
Messages: 193
Registered: June 2005
first-grade participant
From: *physik.uni-giessen.de
Hi Sebastian, yes, I was just about writing the same!!
I completely agree that it is important to keep these events in.
This is real background which will occur in data taking as well.
I actually think this might Bremsstrahlung in the detector material, beampipe or TPC gas (by Geant) with gamma->e+e- and then curling e+/-.
I suspect so because it seems to occur more often in J/Psi->e+e- (about 1/1000 events) than in J/Psi->mu+mu-
(rough estimate is ~1/10000 events or even less).
cheers, Soeren
Re: TPC digitization blocks everything for too many cluster events [message #10808 is a reply to message #10786] Wed, 16 June 2010 18:29 Go to previous messageGo to next message
Malgorzata Gumberidze is currently offline  Malgorzata Gumberidze
Messages: 98
Registered: June 2005
continuous participant
From: *uoks.uj.edu.pl
Hi,

So what i have done is to run simulation (pgun) for positrons with the momentum
range 0.15- 5 GeV/c . For one of the event, e.g. 246, simulations takes 6 minutes.
This is not the worst case of event. Here is printout which i'm getting on the scream.
in addition i'm attaching plot which is showing amplitude of the sample: PndTpcSample.famp()

---------------------------------------------
----- PndTpcDigiPar -----
-- Context/Purpose: PndTpcDefaultContext
---------------------------------------------
PndTpcGasFile: Int_t 0
EField: Double_t 400
AttachFlag: Int_t 0
LogitudinaldiffusionFlag: Int_t 1
TransversediffusionFlag: Int_t 1
DriftDistortionFlag: Int_t 0
zGem: Double_t -40
zMax: Double_t 110
Gain: Double_t 4000
Supression: Double_t 0.001
Spread: Double_t 0.02
MinSignalAmp: Double_t 10
rMin: Double_t 15.5
rMax: Double_t 41.5
PadPlaneFile: Int_t 24
PadShapeFile: Int_t 14
PadShapeRange: Double_t 0.5
PadShapeStep: Double_t 0.02
PadShapeIntStep: Double_t 0.01
ADCThreshold: Int_t 10
ADCMax: Int_t 100000
ADCBits: Int_t 12
SamplingFreq: Double_t 40
T0_wallclock: Double_t -60000
TimeBits: Int_t 28
PSAThreshold: Int_t 100
Shaper_tDiff: Double_t 50
Shaper_tInt: Double_t 50
Shaper_tSig: Double_t 2
TOTPSA: Int_t 1
GAUSSIANNOISE: Int_t 0
GAUSSIANNOISEAMP: Int_t 0
Rate: Double_t 0.02
---------------------------------------------

PndTpcClusterizer:: 18573 clusters created
39515 electrons arriving at readout
Aggregating drifted electrons into avalanches finished.
39515 Avalanches created
0 aggregations done.
73880 Signals created
PndTpcElectronicsTask::Exec
Building up padmap ...finished. 1094 pads hit
...........
1497 Digis created
PndTpcClusterFinderTask::Exec
387 cluster created containing 1497 digis from 1497
Hit array contains 3 hits
PndEmcMakeCluster, event: 1


Hope it will help you, i can also plot different distributions, but it would be nice
if you would be more exact what to plot.

best regards,
gosia
  • Attachment: sample.gif
    (Size: 9.65KB, Downloaded 279 times)
Re: TPC digitization blocks everything for too many cluster events [message #10815 is a reply to message #10789] Tue, 22 June 2010 19:08 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
Dear all,
I think I have found the guilty guy. Which is not tpc digitization, not exactly.
Putting some thousands of cout in PndTpcElectronicTask::Exec, I have found that the code stucks at the following line:

dig->SetLinks(FairMultiLinkedData(kTpcSignal, sigIdMap[padIt->first]));


Now we have understood why the problem was not appearing before: becuase it is related to the new fairlink propagation! If I comment out this line, the code is slow for the "heavy" events, but it is processed in a "reasonable" amount if time...
I have taken my wonderful 5533 event, it took 24 hours with FairLink line, 1 minute without....

I suppose there is something bad in the linking procedure for tpc, for large events, maybe because the too many objects to handle.

Another slow line, but two orders of magnitude faster than the link line, is the following:

 PndTpcDigitizationPolicy().Digitize(sv,&samplelist,ffrontend,fpulseshape);


The code is stucked there for the guilty event, but just for 10 seconds per iteration of the padmap. I am not sure if this is a "feature" or if this could be optimized somehow, I leave the word to digitization experts.

About the SetLinks, I would suggest for the moment to comment out the line, in order to make the digit faster. If somebody wants to check the guilty event, just send me a mail and I could give him the path at GSI where to get a root file with... many problems.
Regards
Re: TPC digitization blocks everything for too many cluster events [message #10816 is a reply to message #10815] Tue, 22 June 2010 20:14 Go to previous messageGo to next message
Felix Boehmer is currently offline  Felix Boehmer
Messages: 149
Registered: May 2007
Location: Munich
first-grade participant

From: *adsl.alicedsl.de
Dear Stefano,

many thanks for the effort you put into this! I didn't look into it yet, but I have to admit I also did not suspect the linking to be the culprit.
Are there any known performance issues with the FairLinks involving any other detectors?

Cheers

Felix
Re: TPC digitization blocks everything for too many cluster events [message #10817 is a reply to message #10816] Tue, 22 June 2010 20:26 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *8-87-r.retail.telecomitalia.it
From what I have seen, all the "handbrakes" were at the tpc level... I should run many events to see if the same effect appears also with other detectors, tomorrow I will try the whole file w/o the SetLink line... and maybe I could tell you.
Re: TPC digitization blocks everything for too many cluster events [message #10818 is a reply to message #10769] Tue, 22 June 2010 20:45 Go to previous messageGo to next message
Jens Sören Lange is currently offline  Jens Sören Lange
Messages: 193
Registered: June 2005
first-grade participant
From: *web.vodafone.de
that's great Stefano. thanks so much.
Re: TPC digitization blocks everything for too many cluster events [message #10819 is a reply to message #10815] Tue, 22 June 2010 22:44 Go to previous messageGo to next message
Malgorzata Gumberidze is currently offline  Malgorzata Gumberidze
Messages: 98
Registered: June 2005
continuous participant
From: *fbx.proxad.net
Hi Ste,

I have commented the same line in my code:
dig->SetLinks(FairMultiLinkedData(kTpcSignal, sigIdMap[padIt->first]));

and now event which before took several hours is fine,
it take maybe 30 seconds.

good job,
gosia
Re: TPC digitization blocks everything for too many cluster events [message #10828 is a reply to message #10816] Thu, 24 June 2010 10:32 Go to previous messageGo to next message
Bjoern Spruck is currently offline  Bjoern Spruck
Messages: 56
Registered: June 2005
Location: Gießen
continuous participant
From: *physik.uni-giessen.de
Felix Boehmer wrote on Tue, 22 June 2010 20:14


Are there any known performance issues with the FairLinks involving any other detectors?



I would expect a similar behavior in Detectors where you have a lot of secondaries. This might be DIRC (with Cherenkov photons on) and EMC. For EMC some cleaning procedure is applied before the linking takes place.
I could comment out that cleaning and check what is happening then.

Bjoern
Re: TPC digitization blocks everything for too many cluster events [message #10829 is a reply to message #10828] Thu, 24 June 2010 13:06 Go to previous message
Tobias Stockmanns is currently offline  Tobias Stockmanns
Messages: 489
Registered: May 2007
first-grade participant
From: *ikp.kfa-juelich.de
Dear colleagues,

I have studied the FairLink problem in more detail. The problem is the enormous amount of data created for the TPC. For one special event it has: 19281 PrimäriClusters, 42958 Drifted Electrons, 42958 Avalanches, 80300 Signals and 1,674,929 Digis.

All this objects have links to the previous one. I could speed up the linking by a factor of 2 by using std::set instead of std::vector. Nevertheless this one event takes 500 s on my PC. I think that for the TPC one should bypass the linking and only connect the TpcClusters with the MC points and only for dedicated studies one switches on linking.

Cheers,

Tobias
Previous Topic: crash in running macro/global/sim_theta.C
Next Topic: radiation length study or G3 vs G4
Goto Forum:
  


Current Time: Sat Dec 07 00:48:42 CET 2024

Total time taken to generate the page: 0.00699 seconds