GSI Forum
GSI Helmholtzzentrum für Schwerionenforschung

Home » Fairroot » General Discussion » PROOF in FairRoot
PROOF in FairRoot [message #13169] Fri, 09 March 2012 22:55 Go to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *dip0.t-ipconnect.de
As of yesterday, 08th March 2012, users have possibility to execute their reconstruction macros on the PROOF clusters.
Please check the website for more instructions and details.
Re: PROOF in FairRoot [message #13281 is a reply to message #13169] Sat, 31 March 2012 14:16 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *gsi.de
Hi radek,
I have tried to run some amcro with proof.

First of all, if I have understood well proof does not work with simulation but only with reconstruction.

I produced a sim file with standard macro/pid/run_sim_sttcombi_pgun.C"(100)", everything fine.
Then I have tried to run the macro/digi/run_digi_sttcombi.C using only the first task putting FairRunAna("proof"), using only the PndSttHitProducerRealFast task.

This is the output:

Toggle Spoiler


It says "finished succesfully" but in reality it has processed nothing, the output file is empty, and there are many errors as you can see.

Then I have tried your example in the website, but it prduces errors the same:

Toggle Spoiler


What is going wrong?

Moreover, one question.
Trying to use different tasks I had several errors, which I have solved simply putting //! for the private objects of the tasks, which however should not be streamed.
But reading your comment on the site I have not well understood if this is safe or no. And if it is not, how to fix it properly?

Re: PROOF in FairRoot [message #13287 is a reply to message #13281] Tue, 03 April 2012 12:28 Go to previous messageGo to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *gsi.de
Dear Stefano,

Sorry for late answer.

The first problem is already solved in the current trunk. The problem was connected with your calling of fRun->Run(0,nEvents) in the macro, where nEvents = 0, which resulted in:

FairRunAna::RunOnProof(): The chain seems to have 100 entries.
FairRunAna::RunOnProof(): There are 1 files in the chain.
FairRunAna::RunOnProof(): Starting inChain->Process("FairAnaSelector","",0,0)

I have fixed it to take the whole chain if the user does not specify number of events to analyze.



The second problem: I guess the code crashes because of problems in MVD/SDS. The thing is I did not fix the classes. I will commit the changes I have implemented there in order to make it running. Sorry for that.


The last question:

"Trying to use different tasks I had several errors, which I have solved simply putting //! for the private objects of the tasks, which however should not be streamed.
But reading your comment on the site I have not well understood if this is safe or no. And if it is not, how to fix it properly?"

I also do it sometimes. There may be only one problem: assume you have some steering variable in your task, like fVerbose. You normally set it in the macro either with a setter or in a dedicated constructor. You would like to have this fVerbose also set in your tasks running on PROOF with the value you specify in the macro. But if you use //!, this value will not be streamed and the default value from the default constructor will be taken. Now fVerbose is not very crucial for running of the tasks but for example in my PndBarrelTrackFinder I have members to choose which detectors to use to search for tracks. I set them in macro and it is very important to pass this information to the tasks on the PROOF nodes. Because of that I cannot just do //! on these members.


yours
radek
Re: PROOF in FairRoot [message #13293 is a reply to message #13287] Wed, 04 April 2012 17:50 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
Now it is also worse, it is crashing:

Toggle Spoiler


I had understood that the example in macro/global was running.
Is it running in your case, and maybe it is only problem of my Ubuntu11.10?
Re: PROOF in FairRoot [message #13297 is a reply to message #13293] Wed, 04 April 2012 21:22 Go to previous messageGo to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *dip0.t-ipconnect.de
Dear Stefano,

As I said before the tasks have to be properly initialized or their members be switched off for streaming ( //! ). This I did not commit to SVN yet. I will do it probably tomorrow when I find some time to do it properly.

I've just got the plain pandaroot trunk and can see the same errors that you have when running with MVD and STT. From my experience I think it is the MVD that causes most of the problems. However when I switch them off, the PROOF does what it should.

So, can you just make a quick check and comment out all the tasks, leaving only GEM digitizer and hit finder?

Below you can see the running with only GEM:

Quote:

karabowi@lxi047::~/pandaroot_15235/trunk/macro/global$ root -l -q 'tracks_BARREL_1000.C("proof",4)' root [0]
Processing tracks_BARREL_1000.C("proof",4)...
FairRootManager::OpenOutFile("tracks_22Part_n4000.root")
[INFO ] The input consists out of the following trees and files:
[INFO ] - cbmsim
[INFO ] - /misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n1000_f0 .root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f1.root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f2.root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f3.root
FairRunAna::RunOnProof(0,4000): running FairAnaSelector on proof server: "" with PAR file name = "$VMCWORKDIR/gconfig/libFairRoot.par".
+++++++ T P R O O F +++++++++++++++++++++++++++++++++
creating TProof* proof = TProof::Open("");
+++ Starting PROOF-Lite with 8 workers +++
Opening connections to workers: OK (8 workers)
Setting up worker servers: OK (8 workers)
PROOF set to parallel mode (8 workers)
+++++++ C R E A T E D +++++++++++++++++++++++++++++++
0309: ClearPackages
0309: UploadPackages
0309: EnablePackages
EXECUTING libFairRoot.par/SETUP.C without includes
Function SETUP_c7827247() busy. loaded after "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C"
Error: G__unloadfile() Can not unload "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C", file busy /tmp/SETUP_c7827247.C:20:
Note: File "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C" already loaded
*** Interpreter error recovered ***
DONT MIND THE ERRORS HERE, ITS EXECUTED.
0309: ShowPackages
*** Package cache client:/misc/karabowi/.proof/packages ***
total 0
drwxr-x--- 3 karabowi had1 22 31. Okt 13:34 libFairRoot
lrwxrwxrwx 1 karabowi had1 60 4. Apr 21:10 libFairRoot.par -> /misc/karabowi/pandaroot_15235/trunk/gconfig/libFairRoot.par

0309: Done
FairRunAna::RunOnProof(): The chain seems to have 4000 entries.
FairRunAna::RunOnProof(): There are 4 files in the chain.
FairRunAna::RunOnProof(): Starting inChain->Process("FairAnaSelector","",4000,0)

Info in <TProofLite::SetQueryRunning>: starting query: 1
Info in <TProofQueryResult::SetRunning>: nwrks: 8
Error in <TGSpeedo::TGSpeedo::Build>: speedo.gif not found
-I- FairAnaSelector::Begin()
Looking up for exact location of files: OK (4 files)
Looking up for exact location of files: OK (4 files)
Info in <TPacketizerAdaptive::TPacketizerAdaptive>: Setting max number of workers per node to 8
Validating files: OK (4 files)
Info in <TPacketizerAdaptive::InitStats>: fraction of remote files 1.000000
-I- FairAnaSelector::Terminate(): fOutput->ls() still sending)
OBJ: TSelectorList TSelectorList Special TList used in the TSelector : 0
OBJ: TList MissingFiles Doubly linked list : 0
OBJ: TStatus PROOF_Status : 0 at: 0x36b60b0
OBJ: TOutputListSelectorDataMap PROOF_TOutputListSelectorDataMap_object Converter from output list to TSelector data members : 0 at: 0x36c33a0
-I- FairAnaSelector::Terminate(): -------------
Lite-0: all output objects have been merged
FairRunAna::RunOnProof(): inChain->Process DONE
------------------------------------------------------------------------ --------
-------------- actual containers in runtime database -------------------------
-------------- runs, versions ------------------------------------------------
run id
container 1st-inp 2nd-inp output
-------------- input/output --------------------------------------------------
first Input:
OBJ: FairParRootFile /misc/karabowi/pandaroot_15235/trunk/macro/global/params_22Part_n1000.ro ot : 0 at: 0x2d9d610
Root file I/O /misc/karabowi/pandaroot_15235/trunk/macro/global/params_22Part_n1000.ro ot is open
detector I/Os: FairGenericParIo
second Input:
Ascii I/O /misc/karabowi/pandaroot_15235/trunk/macro/params/all.par is open
detector I/Os: FairGenericParIo
output: none


Macro finished succesfully.
Output file is tracks_22Part_n4000.root
Parameter file is /misc/karabowi/pandaroot_15235/trunk/macro/global/params_22Part_n1000.ro ot
Real time 14.6925 s, CPU time 0.73 s


And when running with all:

Quote:

karabowi@lxi047::~/pandaroot_15235/trunk/macro/global$ root -l -q 'tracks_BARREL_1000.C("proof",4)' root [0]
Processing tracks_BARREL_1000.C("proof",4)...
FairRootManager::OpenOutFile("tracks_22Part_n4000.root")
Info in (PndGeoHandling::Instance): Making a new instance using the framework.
Info in <PndSdsHybridHitProducer::PndSdsHybridHitProducer>: MVD Hybrid Hit Producer created, Parameters will be taken from RTDB
[INFO ] The input consists out of the following trees and files:
[INFO ] - cbmsim
[INFO ] - /misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n1000_f0 .root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f1.root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f2.root
[INFO ] - file:///misc/karabowi/pandaroot_15235/trunk/macro/global/points_22Part_n 1000_f3.root
FairRunAna::RunOnProof(0,4000): running FairAnaSelector on proof server: "" with PAR file name = "$VMCWORKDIR/gconfig/libFairRoot.par".
+++++++ T P R O O F +++++++++++++++++++++++++++++++++
creating TProof* proof = TProof::Open("");
+++ Starting PROOF-Lite with 8 workers +++
Opening connections to workers: OK (8 workers)
Setting up worker servers: OK (8 workers)
PROOF set to parallel mode (8 workers)
+++++++ C R E A T E D +++++++++++++++++++++++++++++++
0309: ClearPackages
0309: UploadPackages
0309: EnablePackages
EXECUTING libFairRoot.par/SETUP.C without includes
Function SETUP_c7827247() busy. loaded after "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C"
Error: G__unloadfile() Can not unload "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C", file busy /tmp/SETUP_c7827247.C:20:
Note: File "/misc/karabowi/pandaroot_15235/trunk/gconfig/rootlogon.C" already loaded
*** Interpreter error recovered ***
DONT MIND THE ERRORS HERE, ITS EXECUTED.
0309: ShowPackages
*** Package cache client:/misc/karabowi/.proof/packages ***
total 0
drwxr-x--- 3 karabowi had1 22 31. Okt 13:34 libFairRoot
lrwxrwxrwx 1 karabowi had1 60 4. Apr 21:11 libFairRoot.par -> /misc/karabowi/pandaroot_15235/trunk/gconfig/libFairRoot.par

0309: Done
FairRunAna::RunOnProof(): The chain seems to have 4000 entries.
FairRunAna::RunOnProof(): There are 4 files in the chain.
FairRunAna::RunOnProof(): Starting inChain->Process("FairAnaSelector","",4000,0)
Warning in <TBufferFile::WriteObjectAny>: since PndGeoHandling has no public constructor
which can be called without argument, objects of this class
can not be read with the current library. You will need to
add a default constructor before attempting to read it.
Error in <TClass::New>: cannot create object of class FairRuntimeDb
Warning in <TClass::TClass>: no dictionary for class pair<double,FairTimeStamp*> is available
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <TClass::BuildRealData>: Cannot find any ShowMembers function for PndSdsCalcStrip!
Warning in <TClass::TClass>: no dictionary for class BinaryFunctor is available
Error in <TStreamerInfo::Build:>: PndSdsPixelClusterTask: BinaryFunctor* has no streamer or dictionary, data member fFunctor will not be saved
Error in <TStreamerInfo::Build:>: PndSdsPixelClusterTask: BinaryFunctor* has no streamer or dictionary, data member fStartFunctor will not be saved
Warning in <TClass::TClass>: no dictionary for class PndSdsPixelBackMapping is available
Error in <TStreamerInfo::Build:>: PndSdsPixelClusterTask: PndSdsPixelBackMapping* has no streamer or dictionary, data member fBackMapping will not be saved
Warning in <TClass::TClass>: no dictionary for class PndSdsFEAmpModelSimple is available
Error in <TStreamerInfo::Build:>: PndSdsFE: PndSdsFEAmpModelSimple* has no streamer or dictionary, data member fFrontEndModel will not be saved
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <Pair Emulation Building>: const char* is not yet supported in pair emulation
Error in <TStreamerInfo::Build:>: PndSdsStripClusterTask: BinaryFunctor* has no streamer or dictionary, data member fFunctor will not be saved
Error in <TClass::New>: cannot create object of class PndSdsChargeWeightingAlgorithms
Error in <TClass::New>: cannot create object of class PndGeoHandling
Error in <TBufferFile::ReadObject>: could not create object of class PndGeoHandling
Error in <TExMap::Remove>: key 4194304 not found at 291
Warning in <TBufferFile::CheckObject>: reference to object of unavailable class TObject, offset=4194304 pointer will be 0
Error: Symbol #include is not defined in current scope tracks_BARREL_1000.C:117:
Error: Symbol exception is not defined in current scope tracks_BARREL_1000.C:117:
Syntax Error: #include <exception> tracks_BARREL_1000.C:117:
Error: Symbol G__exception is not defined in current scope tracks_BARREL_1000.C:117:
Error: type G__exception not defined FILE:/misc/karabowi/pandaroot_15235/trunk/macro/global/./tracks_BARREL_1 000.C LINE:117
*** Interpreter error recovered ***


I have spent the last few days on fixing the problem with the root parameter files that one gives to FairRuntimeDb. Previously one needed to store the parameters in one root file while doing simulation. Currently FairParRootFileIo::open is taking also a TList of root parameter file names. Internally the files are merged into one file that is then given as input to the FairRuntimeDb as parameter file.

This you can test with sim_BARREL_1000sep.C and tracks_BARREL_1000sep.C

yours
radek

[Updated on: Wed, 04 April 2012 21:24]

Report message to a moderator

Re: PROOF in FairRoot [message #13310 is a reply to message #13297] Thu, 12 April 2012 09:50 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
Hi,
I have just tried the new "sep" macros.
I have used only 1 input file, I had to modify the parameter part because it was searching for 4 parameter files while I had only one. It seems it is running:

Toggle Spoiler


but the output file tracks_22Part_n1000.root is empty. In the folder I have one tracks_22Part_n1000_worker_0.3.root file which seems properly filled.
What is going wrong?

If I try with 10 files, adding 10 paramater files in the list, I have:

Toggle Spoiler


Maybe I have messed up something.
Re: PROOF in FairRoot [message #13374 is a reply to message #13310] Thu, 19 April 2012 16:23 Go to previous messageGo to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *gsi.de
Dear Stefano,

Again it took me long time to answer, but:
a/ on GSI Debian the merging was working without any problems
b/ on my MAC the merging was crashing badly with no clue as to where the crash happens.

After few days of trying to find the problem, I've used my original idea of merging the files by myself. With the current changes (rev. 15330) the merging works both on MAC and Debian, but it's slower than the original ROOT TFlieMerger.

Anyways, can you try now?

yours
radek

ps. The "merging" somehow worker for 1 file, that's why it was analyzed. Apparently only on one worker.

The standard output file (like "tracks_22Part_n1000.root") is empty, sorry for that. It is created when the master FairRunAna is created. Originally I though of using it for creating a merged file, but since file merging crashes, it just stays empty. I know it's a bit annoying. If you really think it should not be created, please let me know.
Re: PROOF in FairRoot [message #13426 is a reply to message #13374] Thu, 03 May 2012 10:17 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
I had the time to check the new updates.
Now it seems the system runs, with 10 files and also with a single file. In the latter case I hope all the workers are used, considering that at the end I have only one final root file (worker_0.1) and not 8 as I would expect.

Now it would be good to have it running also for detectors different from GEM. In this sense we could "program" updates in the code, made by the package responsible, so that all the classes are "clean" and running. Who could take care?
Re: PROOF in FairRoot [message #13428 is a reply to message #13426] Fri, 04 May 2012 00:11 Go to previous messageGo to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *dip0.t-ipconnect.de
Dear Stefano,

Thank you for again testing PROOF. I have two comments to your last message:

- the "one input tree per one worker" seems to be a part of the adaptive PacketizerStrategy. When changing the strategy to the old one, 1 file is analyzed on several workers (but I had to introduce some sleep in exec to see it).
I have reorganized slightly the FairRunAna, so that one can set proof parameters from the outside as well.
Please update the trunk and add the following line to your macro, somewhere between new FairRunAna and fRun->Run():

 fRun->GetProof()->SetParameter("PROOF_PacketizerStrategy", (Int_t)0);


- another tasks - I am working on it. Currently I am testing the changes implemented on linux debian before last PandaCM on my MAC. I hope to be ready tomorrow - after commit will let you know.

yours
radek
Re: PROOF in FairRoot [message #13430 is a reply to message #13426] Sat, 05 May 2012 12:07 Go to previous messageGo to next message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *dip0.t-ipconnect.de
Dear Stefano et al,

I have just committed the changes necessary to run my proof macros (macro/global/{sim,tracks}_BARREL_1000sep.C) with MVD, STT, GEM and my track finding. The changes are located in files:

M mvd/MvdReco/PndMvdStripClusterTask.cxx
M mvd/MvdDigi/PndMvdNoiseProducer.cxx
M sds/SdsReco/PndSdsStripClusterTask.cxx
M sds/SdsReco/PndSdsChargeWeightingAlgorithms.cxx
M sds/SdsReco/PndSdsPixelClusterTask.cxx
M sds/SdsReco/PndSdsStripClusterTask.h
M sds/SdsReco/PndSdsChargeWeightingAlgorithms.h
M sds/SdsReco/PndSdsPixelClusterTask.h
M sds/SdsDigi/PndSdsHybridHitProducer.h
M sds/SdsDigi/PndSdsStripHitProducer.cxx
M sds/SdsDigi/PndSdsNoiseProducer.cxx
M sds/SdsDigi/PndSdsHybridHitProducer.cxx
M sds/SdsDigi/PndSdsStripHitProducer.h
M PndTools/generalTools/PndGeoHandling.cxx
M PndTools/generalTools/PndGeoHandling.h
M stt/PndSttHitProducerIdeal.cxx
M macro/global/tracks_BARREL_1000sep.C

Sometimes it's more, sometimes less. Usually it's only member initialization in the class constructors, but I've also had some fun with PndGeoHandling* fGeoH and PndSttMapCreator* mapper. If any of the people responsible for the classes read this message, please do have a look at the changes, so that I do not destroy anything.

BTW, In the file attached to this message, I've tried to summarize my thought processes, when dealing with the PROOF crashes, and the changes I had to introduce to MVD/STT/Global. How you will find it useful.

yours
radek
Re: PROOF in FairRoot [message #13436 is a reply to message #13430] Mon, 07 May 2012 13:54 Go to previous messageGo to next message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
Hi Radek,
one month ago I had already a discussion with Tobias about PndGeoHandling public constructor, and it should be private as long as PndGeoHandling is a singleton. Otherwise it could happen that it is called by a different subprocess.
No idea on how to solve this point.

Re: PROOF in FairRoot [message #13510 is a reply to message #13436] Tue, 22 May 2012 16:49 Go to previous message
Radoslaw Karabowicz is currently offline  Radoslaw Karabowicz
Messages: 108
Registered: June 2004
Location: GSI
continuous participant
From: *gsi.de
Is it possible to check in the constructor for the existence of the PndGeoHandling and crash in the positive case?

yours
radek
Previous Topic: Getting Hits by detectors registered in the output root file
Next Topic: Detector response analysis
Goto Forum:
  


Current Time: Tue Apr 16 12:16:59 CEST 2024

Total time taken to generate the page: 0.00863 seconds