Home » PANDA » PandaRoot » Tracking » Benchmark Tables and plots for tracking performance
| Benchmark Tables and plots for tracking performance [message #11107] |
Tue, 19 October 2010 22:18  |
Gianluigi Boca
Messages: 177
Registered: March 2004
|
first-grade participant |
From: *gsi.de
|
|
Dear all,
with this rather long message, following up the discussion at the last EVO meeting, I would like to suggest some Tables
and plots to use for benchmark of a global tracker
algorithm(s).
1- Benchmark Table concerning hits and tracks.
For this type of Benchmark, only the Pattern Recognition is necessary, not the subsequent track fitting.
I would generate tracks with 5 degrees< Theta < 120 degrees, in the full Fi range, with the Box Generator.
The vertex could be randomly distributed at +/- 0.4 cm from (0,0,0).
The benchmark procedure would consist of :
a) Generation of events with 1 track, 5 tracks, 10 tracks at different total momenta with the Box Generator to check the performance at different phase space regions.
For instance :
at 0.1 GeV/c 1 track, 5 tracks, 10 tracks
at 0.3 GeV/c 1 track, 5 tracks, 10 tracks
at 1. GeV/c 1 track, 5 tracks, 10 tracks
at 5. GeV/c 1 track, 5 tracks
at 10. GeV/c 1 track, 2 tracks
b) Exclusion from analysis of montecarlo tracks producing
less than a predefined number (for instance : 3) hits in the Panda central detector. Also exclusion of those MC tracks not originating from the interaction vertex.
c) Assignment of each track found by the global Pattern Recognition to one (if possible) MC track.
(Naturally and unfortunately there is no unique way of assigning a track found by PR to the corresponding MC track. See below the method I suggest to use).
d) COMPUTATION OF A TABLE WITH THE % OF TRUE HITS FOUND,
THE NUMBER OF HITS NOT BELONGING (SPURIOUS); IF A TRACK IS NOT ASSIGNED TO ANY MC TRACK, increase the counter of spurious tracks. IF A MC TRACK HAS NO PR TRACK ASSIGNED, increase the counter of unmatched tracks.
This numbers are the Benchmark Table.
2- Benchmark Plots, concerning momentum resolutions and physics
channels performance.
For these plots PR + momentum fitting (Kalman with Genfit) is needed.
a) Plot of the residual of total momentum for all momenta and track topologies generated as said before.
In other words : plot of
[(Total Reconstructed Momentum) - (MC truth Momentum)]/
(Error on Total Reconstructed Momentum)
b) Plot of the residual of Pper (Momentum in XY plane) for each momentum and all topologies.
c) Plot of some typical physical EXCLUSIVE channel, with all charged particles. An examples could be :
At lower energy :
- f_2(2000) --> fi fi --> K+K-K+K-
Intermediate energy, only 2 tracks :
- J/Psi --> mu+ mu- ;
- J/Psi --> pi+ pi- ;
Higher energy, 6 tracks, more complicated topology :
Y(4320) --> psi_2S(3686) pi+pi-
|
|--> J/Psi (mu+mu-) pi+pi-
so actually this is Y(4320) --> pi+pi+pi-pi-mu+mu-
OF THESE CHANNEL THE BENCHMARK NUMBERS COULD BE :
- # OF EVENTS WITH MASS 'CLOSE' TO THE NOMINAL ONE, I.E. # EVENTS RECONSTRUCTED;
- THE RESOLUTION of the primary generated particle mass.
Appendix
SCHEME TO ASSIGN PATTERN RECOGNITION FOUND TRACKS TO MC TRACKS.
There is no unique way of assigning a track found by PR to the corresponding MC track.
Here I propose a simple scheme that works in the general case in which tracks are found by PR, with hits belonging to different MC tracks :
a) first calculate a 2-dimensional Matrix ; the number of rows are equal to the # of tracks found by PR, the # of columns is equal to the # of MC tracks having at least 1 hit in any of the tracks found by PR;
each cell of this Matrix is filled with the corresponding number of hits.
For instance, suppose that the track # 2 found by PR has 7 hits
belonging to MC track # 5, that the cell (2,5) is filled with 7.
Do the filling looping over all tracks found by PR.
b) Find the cell in the Matrix with the largest number. Assign the corresponding track found to PR to the MC track.
For instance, suppose the cell (5 , 8) contains 25, and that is the maximum value in the whole Matrix. The the track found # 5 is assigned to the MC track #8.
Erase the row and column just matched ( row 5, column 8 in the example)
c) Consider the remaining Matrix; iterate the procedure in b) and assign another track found by PR to a MC track. Iterate untill ALL MC TRACKS or ALL TRACK FOUND BY PR have been exhausted.
Gianluigi
|
|
|
|
| Re: Benchmark Tables and plots for tracking performance [message #11124 is a reply to message #11107] |
Wed, 27 October 2010 09:09  |
StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
|
first-grade participant |
From: *vpn.unito.it
|
|
With some delay here you are my comments to Gianluigi proposal.
| Quote: |
I would generate tracks with 5 degrees< Theta < 120 degrees, in the full Fi range, with the Box Generator.
The vertex could be randomly distributed at +/- 0.4 cm from (0,0,0).
|
I would prefer to separate effects coming from displaced vertex. I think it would be better to run IP (0,0,0), and only in a second time with some vertex smearing.
| Quote: |
The benchmark procedure would consist of :
a) Generation of events with 1 track, 5 tracks, 10 tracks at different total momenta with the Box Generator to check the performance at different phase space regions.
|
Fine for me.
| Quote: |
For instance :
at 0.1 GeV/c 1 track, 5 tracks, 10 tracks
at 0.3 GeV/c 1 track, 5 tracks, 10 tracks
at 1. GeV/c 1 track, 5 tracks, 10 tracks
at 5. GeV/c 1 track, 5 tracks
at 10. GeV/c 1 track, 2 tracks
|
Not fine for me. The low momentum region has performances more sensitive to momentum, and it needs a finer check. I would propose a flat distribution from 0.1 to 2 GeV, and then two more points 5 - 10 GeV, or 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.5 2 5 10.
| Quote: |
b) Exclusion from analysis of montecarlo tracks producing
less than a predefined number (for instance : 3) hits in the Panda central detector. Also exclusion of those MC tracks not originating from the interaction vertex.
|
Maybe you mean exclusion of reconstructed tracks with less than the number. Or you want to separate acceptance from reco efficiency? I would consider only primary particles.
| Quote: |
c) Assignment of each track found by the global Pattern Recognition to one (if possible) MC track.
(Naturally and unfortunately there is no unique way of assigning a track found by PR to the corresponding MC track. See below the method I suggest to use).
d) COMPUTATION OF A TABLE WITH THE % OF TRUE HITS FOUND,
THE NUMBER OF HITS NOT BELONGING (SPURIOUS); IF A TRACK IS NOT ASSIGNED TO ANY MC TRACK, increase the counter of spurious tracks. IF A MC TRACK HAS NO PR TRACK ASSIGNED, increase the counter of unmatched tracks.
This numbers are the Benchmark Table.
|
I have not well understood your method, could you please do an example? I don't think there could be reco tracks without mc assignment (apart matching a series of mvd noise hits...). And you have not said when you assign a mc id to a particle (in each case, the most common mc id? only with at least 60% of mc id? what?) [Ok, I have seen there are some explanations at the end]
| Quote: |
2- Benchmark Plots, concerning momentum resolutions and physics
channels performance.
For these plots PR + momentum fitting (Kalman with Genfit) is needed.
|
We need also momentum after prefit, without kalman. This tells us what the kalman is really doing.
| Quote: |
a) Plot of the residual of total momentum for all momenta and track topologies generated as said before.
In other words : plot of
[(Total Reconstructed Momentum) - (MC truth Momentum)]/
(Error on Total Reconstructed Momentum)
|
This tells you only the quality of your fit, it should centred at zero with a "1" sigma. You need also as a function of theta?
| Quote: |
b) Plot of the residual of Pper (Momentum in XY plane) for each momentum and all topologies.
|
and theta.
I think we need momentum residual and mom resolution as a function of momentum and theta, theta residual and resolution, and efficiency also for prefitting and fitting (how to define this efficiency? within 5 sigma?). For each particle multiplicity.
| Quote: |
c) Plot of some typical physical EXCLUSIVE channel, with all charged particles. An examples could be :
|
I think this comes later. First we need to validate reconstruction apart from physics channels. We will need them in a second time.
| Quote: |
a) first calculate a 2-dimensional Matrix ; the number of rows are equal to the # of tracks found by PR, the # of columns is equal to the # of MC tracks having at least 1 hit in any of the tracks found by PR;
each cell of this Matrix is filled with the corresponding number of hits.
For instance, suppose that the track # 2 found by PR has 7 hits
belonging to MC track # 5, that the cell (2,5) is filled with 7.
Do the filling looping over all tracks found by PR.
b) Find the cell in the Matrix with the largest number. Assign the corresponding track found to PR to the MC track.
For instance, suppose the cell (5 ,  contains 25, and that is the maximum value in the whole Matrix. The the track found # 5 is assigned to the MC track #8.
Erase the row and column just matched ( row 5, column 8 in the example)
c) Consider the remaining Matrix; iterate the procedure in b) and assign another track found by PR to a MC track. Iterate untill ALL MC TRACKS or ALL TRACK FOUND BY PR have been exhausted.
|
Could you please provide a standard macro with such kind of code, that starting from PndMCTrack TCA and PndTrackCand TCA is able to do this job and fill the corresponding tables/plots? Probably you have it already, done for STT. I need some additional time to think about this methid, but I think it could be a good starting method.
It should be the same for all the tracking codes, then we don't have to repeat the same job (I mean, writing the code) many times (PndRecoQATask ?). You have clearer and smarter idea on the subject than me. In this case, the tracking developer should only produce the statistics, filling the table and the plots, and cross checking what is going wrong, and we introduce less sistematics from "buggy code" ... 
Maybe, writing a global reco qa task could be even better, so that we can use it for dashboard; I like this idea.
This discussion is open to everybody, and other comments are welcome!
|
|
|
|
| Re: Benchmark Tables and plots for tracking performance [message #11158 is a reply to message #11124] |
Tue, 02 November 2010 17:59 |
Gianluigi Boca
Messages: 177
Registered: March 2004
|
first-grade participant |
From: *gsi.de
|
|
Hi Stefano,
thanks for your reply. I generally agree with your suggestions.
In this message I only discuss those points where I have some more comments.
| Stefano Spataro wrote on Wed, 27 October 2010 09:09 |
Maybe you mean exclusion of reconstructed tracks with less than the number. Or you want to separate acceptance from reco efficiency? I would consider only primary particles.
|
I mean that I would consider only tracks leaving a number of hits in the central tracker ENOUGH TO BE RECONSTRUCTED via software (a reasonable number is 3 hits for
a Helix ). That would exclude for instance the tracks falling in the target pipe vertical gap which are not very interesting for our porposes.
| Quote: |
and theta.
I think we need momentum residual and mom resolution as a function of momentum and theta, theta residual and resolution, and efficiency also for prefitting and fitting (how to define this efficiency? within 5 sigma?). For each particle multiplicity.
|
OK ; for the efficiency, we could use 3 sigma from the central value of the peak.
| Quote: |
I think this comes later. First we need to validate reconstruction apart from physics channels. We will need them in a second time.
|
I disagree on this particular point. I believe that a check with a physical channel gives an immediate feeling on the algorithm is working; besides, the cross check with a physics channel is the ultimate benchmark necessary for any tracking.
| Quote: |
Could you please provide a standard macro with such kind of code, that starting from PndMCTrack TCA and PndTrackCand TCA is able to do this job and fill the corresponding tables/plots?
|
As Obama said : "Yes, we can " and I will be happy to provide such a Macro soon if people ask me to.
Cheers Gianluigi
|
|
|
|
Goto Forum:
Current Time: Sun Jul 28 18:23:23 CEST 2024
Total time taken to generate the page: 0.00654 seconds
|
GSI Forum
Search
Help
Members
Login
Home
