GSI Forum - RDF feed
type: channel
description: GSI Helmholtzzentrum für Schwerionenforschung
items: Out of memory problem in EmcPoint - FairLink ?
items: Re: Out of memory problem in EmcPoint - FairLink ?
items: Out of memory problem in EmcPoint - FairLink !
items: Re: Out of memory problem in EmcPoint - FairLink !

Out of memory problem in EmcPoint - FairLink ?
type: item
description: Dear all, trying to run many DPM events we have realized that sometimes the simulation (run_sim_tpccombi_dpm.C) crashes, sometime with a "bad alloc" problem, sometime with a #include error, and without any helpful messages. In order to check what is really going wrong, I have run the vsame macro under valgrind and finally, after waiting 15 hours, I have found the guilty guy, or at least one of them. First of all, these are the dpm settings: PndDpmDirect initialization Momentum = 15 Seed = 4357 Mode = 1 Theta min = 0.001 At the beginning I have many valgrind messages, unitialized values in writing the parameter containers, maybe some expert (Mohammad) should take a look there: Toggle Spoiler -I- FairMCApplication:: Monte carlo Engine Initialisation with TGeant3TGeo RuntimeDb: write container PndEmcGeoPar *** PndEmcGeoPar written to ROOT file version: 1 RuntimeDb: write container PndSensorNamePar *** PndSensorNamePar written to ROOT file version: 1 RuntimeDb: write container PndEmcDigiPar *** PndEmcDigiPar written to ROOT file version: 1 RuntimeDb: write container PndEmcDigiNonuniformityPar RuntimeDb: write container FairBaseParSet ==8505== Use of uninitialised value of size 4 ==8505== at 0x42169AF: deflate_fast (in /home/spataro/jan10/tools/root/lib/li bCore.so.5.26) ==8505== by 0x421556D: deflate (in /home/spataro/jan10/tools/root/lib/libCore .so.5.26) ==8505== by 0x42127FA: R__zip (in /home/spataro/jan10/tools/root/lib/libCore. so.5.26) ==8505== by 0x5F06617: TKey::TKey(TObject const*, char const*, int, TDirector y*) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==8505== by 0x5EEDC88: TFile::CreateKey(TDirectory*, TObject const*, char con st*, int) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==8505== by 0x5EE2F60: TDirectoryFile::WriteTObject(TObject const*, char cons t*, char const*, int) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==8505== by 0x41520E7: TObject::Write(char const*, int, int) const (in /home/ spataro/jan10/tools/root/…
creator: StefanoSpataro
date: 2010-08-03T22:15:03-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Trying to understand what is making our memory explode, under suggestion of Ralf I have run only one event by valgrind. This is the final log message, with all the "problems": Toggle Spoiler RealTime=1391.664607 seconds, CpuTime=1312.570000 seconds (int)58 root [1] .q ==27961== ==27961== ERROR SUMMARY: 32858 errors from 24 contexts (suppressed: 113902 from 4) ==27961== malloc/free: in use at exit: 123,231,441 bytes in 791,157 blocks. ==27961== malloc/free: 11,175,726 allocs, 10,384,569 frees, 1,307,048,594 bytes allocated. ==27961== For counts of detected errors, rerun with: -v ==27961== searching for pointers to 791,157 not-freed blocks. ==27961== checked 140,163,608 bytes. ==27961== ==27961== ==27961== 2 bytes in 1 blocks are definitely lost in loss record 5 of 401 ==27961== at 0x4004405: malloc (vg_replace_malloc.c:149) ==27961== by 0x47A1685: G__memfunc_para_setup (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x47A1DD1: G__parse_parameter_link (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x47A9E22: G__memfunc_setup_imp (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x47AA779: G__memfunc_setup (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x4D859C4: G__cpp_setup_func3() (in /home/spataro/jan10/tools/root/lib/libMathCore.so.5.26) ==27961== by 0x4D85D99: G__cpp_setup_funcG__MathCore (in /home/spataro/jan10/tools/root/lib/libMathCore.so.5.26) ==27961== by 0x4D86D91: G__cpp_setupG__MathCore (in /home/spataro/jan10/tools/root/lib/libMathCore.so.5.26) ==27961== by 0x4782AE4: G__call_setup_funcs (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x4782DE5: G__main (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x4784BE3: G__init_cint (in /home/spataro/jan10/tools/root/lib/libCint.so.5.26) ==27961== by 0x41F5FD8: TCint::ResetAll() (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==27961== ==27961== ==27961== 18 bytes in 3 blocks are p…
creator: StefanoSpataro
date: 2010-08-05T14:42:05-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi Stefano, Quote: PndStack::UpdateTrackIndex(TRefArray*) (PndStack.cxx:332) TIterator* detIter = detList->MakeIterator(); solved, delete detIter is added to the code. Quote: FairParRootFileIo::open(char const*, char const*, char const*, int) (FairParRootFileIo.cxx:103) -> fstream *f = new fstream(fname); solved, delete f is added at the right place. Quote: FairModule::ExpandNode(TGeoNode*) (FairModule.cxx:293) -> M->SetDefaultName(); Here I am not sure! the method (M->SetDefaultName()) is doing much more than setting a name, it check the type of transformation, prepend a letter corresponding to the type and finally append the index of the transformation to the current GeoManager. and in any case we are speaking about 100 kB per session. So I do not believe that this is a big problem, I agree it has to be solved but we still have much worse problems. regards Mohammad
creator: Mohammad Al-Turany
date: 2010-08-06T07:47:54-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Mohammad wrote Quote: FairModule::ExpandNode(TGeoNode*) (FairModule.cxx:293) -> M->SetDefaultName(); Here I am not sure! the method (M->SetDefaultName()) is doing much more than setting a name, it check the type of transformation, prepend a letter corresponding to the type and finally append the index of the transformation to the current GeoManager. and in any case we are speaking about 100 kB per session. So I do not believe that this is a big problem, I agree it has to be solved but we still have much worse problems. Hi, I think we just could copy this function and change the way of setting the name. This would look like that: void FairModule::SetDefaultMatrixName(TGeoMatrix* matrix) { // Copied from root TGeoMatrix::SetDefaultName() and modified (memory leak) // If no name was supplied in the ctor, the type of transformation is checked. // A letter will be prepended to the name : // t - translation // r - rotation // s - scale // c - combi (translation + rotation) // g - general (tr+rot+scale) // The index of the transformation in gGeoManager list of transformations will // be appended. if (!gGeoManager) return; if (strlen(matrix->GetName())) return; char type = 'n'; if (matrix->IsTranslation()) type = 't'; if (matrix->IsRotation()) type = 'r'; if (matrix->IsScale()) type = 's'; if (matrix->IsCombi()) type = 'c'; if (matrix->IsGeneral()) type = 'g'; TObjArray *matrices = gGeoManager->GetListOfMatrices(); Int_t index = 0; if (matrices) index =matrices->GetEntriesFast() - 1; matrix->SetName(Form("%c%i", type, index)); } Kind regards, Ralf.
creator: Ralf Kliemt
date: 2010-08-06T07:57:56-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi Ralf, Ok, it is in SVN now! Mohammad
creator: Mohammad Al-Turany
date: 2010-08-06T10:23:57-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Thanks Mohammad, I have updated svn and run valrind one a single event. Now, what is left is, in the initialization: Toggle Spoiler RuntimeDb: write container FairBaseParSet ==7891== Use of uninitialised value of size 4 ==7891== at 0x42169AF: deflate_fast (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x421556D: deflate (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x42127FA: R__zip (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x5F06617: TKey::TKey(TObject const*, char const*, int, TDirectory*) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==7891== by 0x5EEDC88: TFile::CreateKey(TDirectory*, TObject const*, char const*, int) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==7891== by 0x5EE2F60: TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==7891== by 0x41520E7: TObject::Write(char const*, int, int) const (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x4151F62: TObject::Write(char const*, int, int) (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x7EDA4C1: FairDetParRootFileIo::write(FairParSet*) (FairDetParRootFileIo.cxx:70) ==7891== by 0x7EDE7E5: FairParGenericSet::write(FairParIo*) (FairParGenericSet.cxx:43) ==7891== by 0x7EE8E6A: FairRuntimeDb::writeContainer(FairParSet*, FairRtdbRun*, FairRtdbRun*) (FairRuntimeDb.cxx:297) ==7891== by 0x7EE8A37: FairRuntimeDb::writeContainers() (FairRuntimeDb.cxx:242) ==7891== ==7891== Use of uninitialised value of size 4 ==7891== at 0x42169C6: deflate_fast (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x421556D: deflate (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x42127FA: R__zip (in /home/spataro/jan10/tools/root/lib/libCore.so.5.26) ==7891== by 0x5F06617: TKey::TKey(TObject const*, char const*, int, TDirectory*) (in /home/spataro/jan10/tools/root/lib/libRIO.so.5.26) ==7891==…
creator: StefanoSpataro
date: 2010-08-06T11:02:52-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi Stefano, I tried to do something with PndEmcStructure, by adding at the end of PndEmcStructure::crystal_name_analysis() delete subStrL; where TObjArray *subStrL = TPRegexp("^cave/Fsc_(\\d+)/emc(\\d+)r(\\d+)c(\\d+)_0$").MatchS(node_path); But it somehow only increased memory consumption and calculation time. So I suspect this code should be rewriten somehow without regular expressions. But it will take some time and I can try to do it in two weeks after my vacations. Best regards, Dima
creator: Dima Melnychuk
date: 2010-08-06T12:57:41-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi all, Since Stefano reported memory leaks caused by regular expressions - TPRegexp in PndEmcStructure, I have replaced it by sscanf (rev. 9846). However I didn't manage to check how it affected memory leaks by myselfe. Running valgrind --tool=cachegrind root -b run_sim_tpccombi_dpm.C I do not have such an informative output as reported by Stefano but just DIGI EXECUTION ********************* RealTime=247.386149 seconds, CpuTime=233.050000 seconds (int)56 root [1] .q ==5031== ==5031== I refs: 2,517,267 ==5031== I1 misses: 2,578 ==5031== L2i misses: 1,392 ==5031== I1 miss rate: 0.10% ==5031== L2i miss rate: 0.05% ==5031== ==5031== D refs: 1,291,254 (927,120 rd + 364,134 wr) ==5031== D1 misses: 40,211 ( 37,743 rd + 2,468 wr) ==5031== L2d misses: 6,443 ( 5,450 rd + 993 wr) ==5031== D1 miss rate: 3.1% ( 4.0% + 0.6% ) ==5031== L2d miss rate: 0.4% ( 0.5% + 0.2% ) ==5031== ==5031== L2 refs: 42,789 ( 40,321 rd + 2,468 wr) ==5031== L2 misses: 7,835 ( 6,842 rd + 993 wr) ==5031== L2 miss rate: 0.2% ( 0.1% + 0.2% ) I suspected that it was due to external packages compiled without debug info, but I recomplied it with debug info and it didn't help. Maybe I don't run valgrind with proper flag? Dima
creator: Dima Melnychuk
date: 2010-09-13T13:25:58-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi Dima You have to do valgrind root.exe run_sim_tpccombi_dpm.C Yohave to call valgrind with root.exe which is an executable whereas root is only a wrapper script. So you tested only for memory problems in the wrapper script. Ciao Florian
creator: Florian Uhlig
date: 2010-09-13T13:57:56-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi Dima. I suggest to try the option --lek-check=full since you're hunting memory leaks. Greetings, Ralf.
creator: Ralf Kliemt
date: 2010-09-13T14:11:00-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: I am using the following: valgrind --leak-check=full --suppressions=/home/spataro/jan10/tools/root/etc/valgrind-root.supp root.exe In some versions you should use "suppression" instead of "suppressions", to disable the printout of all the well known root leakages.
creator: StefanoSpataro
date: 2010-09-14T10:48:01-00:00

Out of memory problem in EmcPoint - FairLink !
type: item
description: Finally I have found who is making the analysis crash. I would like to repeat, the problem occurs trying to run many dpm events, i.e. 10000. You can find two kinds of crashes: terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc or Error: Symbol #include is not defined in current scope run_sim_tpccombi_dpm.C:141: Error: Symbol exception is not defined in current scope run_sim_tpccombi_dpm.C:141: Syntax Error: #include <exception> run_sim_tpccombi_dpm.C:141: Error: Symbol G__exception is not defined in current scope run_sim_tpccombi_dpm.C:141: Error: type G__exception not defined FILE:/d/panda02/spataro/pandaroot/macro/pid/64/2/./run_sim_tpccombi_dpm.C LINE:141 *** Interpreter error recovered *** terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc After a lot of tests, disabling and enabling detectors, putting cout and using valgrind, I have find that the main reason of these leaks is explained by valgrind by the following message: ==18644== 10,028,992 bytes in 156,703 blocks are definitely lost in loss record 358 of 361 ==18644== at 0x4A20929: operator new(unsigned long) (vg_replace_malloc.c:230) ==18644== by 0x12800BAB: __gnu_cxx::new_allocator<std::_Rb_tree_node<FairLink> >::allocate(unsigned long, void const*) (new_allocator.h:92) ==18644== by 0x12800BCF: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>, std::less<FairLink>, std::allocator<FairLink> >::_M_get_node() (stl_ tree.h:357) ==18644== by 0x12800C35: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>, std::less<FairLink>, std::allocator<FairLink> >::_M_create_node(Fair Link const&) (stl_tree.h:366) ==18644== by 0x12800D4E: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>, std::less<FairLink>, std::allocator<FairLink> >::_M_insert_(std::_Rb _tree_node_base const*, std::_Rb_tree_node_base const*, FairLink const&) (stl_tree.h:852) ==18644== by 0x12800EB3: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>…
creator: StefanoSpataro
date: 2010-09-14T11:05:45-00:00

Re: Out of memory problem in EmcPoint - FairLink !
type: item
description: Hi Stefano I also fear that there is a problem with the FairLinks or the way of usage. Whenruning one of the Cbm Macros on the GSI batch farm i end up with a segmentation violation always after around 350 Events. After the third crash i monitored the memory consumption and this was always increasing. When the memory consumption reached the limit of 4GB on the batch farm machines and terminates. Checking the process with valgrind for some events I found exactly the same problem as you did. The number of lost bytes was increasing with increasing number of events. ==20123== 7,844,864 bytes in 122,576 blocks are still reachable in loss record 511 of 514 ==20123== at 0x4A1C929: operator new(unsigned long) (vg_replace_malloc.c:230) ==20123== by 0xE9F04F3: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>, std::less<FairLink>, std::allocator<FairLink> >::_M_insert(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, FairLink const&) (new_allocator.h:88) ==20123== by 0xE9F0638: std::_Rb_tree<FairLink, FairLink, std::_Identity<FairLink>, std::less<FairLink>, std::allocator<FairLink> >::insert_unique(FairLink const&) (stl_tree.h:931) ==20123== by 0xE9F0D5A: FairMultiLinkedData::InsertLink(FairLink) (stl_set.h:321) ==20123== by 0xE9F2BA6: FairMultiLinkedData::AddLink(FairLink, bool, float) (FairMultiLinkedData.cxx:123) ==20123== by 0xEDB17EE: CbmStsDigi::CbmStsDigi(int, int, int, int, int, int, int) (CbmStsDigi.h:63) ==20123== by 0xF8E92CD: CbmStsIdealDigitize::Exec(char const*) (CbmStsIdealDigitize.cxx:186) ==20123== by 0x4D83402: TTask::ExecuteTasks(char const*) (TTask.cxx:312) ==20123== by 0x4D83691: TTask::ExecuteTask(char const*) (TTask.cxx:275) ==20123== by 0xEA06EAA: FairRunAna::Run(int, int) (FairRunAna.cxx:279) ==20123== by 0xEA1CC1C: G__FairDict_792_0_5(G__value*, char const*, G__param*, int) (FairDict.cxx:11420) ==20123== by 0x559C3D0: Cint::G__ExceptionWrapper(int (*)(G__value*, char const*, G__param*, int), G__value*, char*, G__param*, int) (Api.cxx:385) =…
creator: Florian Uhlig
date: 2010-09-14T13:50:12-00:00

Re: Out of memory problem in EmcPoint - FairLink !
type: item
description: Hi all, I try to figure out where the problem is. Unfortunately I do not see a potential problem in the code. I do not use pointers and I do not use any "new" in the code. Maybe it is a problem with std::set I use as container to store the data. I will check if I can replace it with either std::vector or a root class. Cheers, Tobias
creator: Tobias Stockmanns
date: 2010-09-14T15:02:51-00:00

Re: Out of memory problem in EmcPoint - FairLink !
type: item
description: Hi Tobias Quote: I try to figure out where the problem is. Unfortunately I do not see a potential problem in the code. I do not use pointers and I do not use any "new" in the code. This was exactely my problem when I looked in the code. I don't see any potential problem. Quote: Maybe it is a problem with std::set I use as container to store the data. I will check if I can replace it with either std::vector or a root class. Why did you choose to use a set instead of a vector? What are the advantages of using a set? Ciao Florian
creator: Florian Uhlig
date: 2010-09-14T15:34:16-00:00

Re: Out of memory problem in EmcPoint - FairLink !
type: item
description: Hi Florian, a set is like a map where only the key is present. This means that you do not have twice the same entry and the data is sorted. This speeds up the find method quite a lot which is important for large link collections like in the TPC and EMC. Ciao, Tobias
creator: Tobias Stockmanns
date: 2010-09-15T06:58:17-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Dear pandaRooters, it seems that I have found the solution for the memory leak of FairMultiLinkedData. The problem is coming from the TClonesArray. Whenever you call ->Clear() for a TClonesArray which contains FairMultiLinkedData you have to substitute it with ->Delete(). After this the memory leak is not visible anymore in Valgrind and Totalview. Please test it and tell me your results. Cheers, Tobias
creator: Tobias Stockmanns
date: 2010-09-30T10:45:10-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Ehm, please give me one week (quite busy at the moment, fighting against another crash)
creator: StefanoSpataro
date: 2010-09-30T12:24:36-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi all, Following the discussion started by Tobias today on EVO meeting concerning memory allocation by TClonesArray I want to propose an idea to use TClonesArray::Compress() (which remove empty slots) if TClonesArray::Capacity() (i.e. number of allocated slots) is above certain threshold, and this threshold can be subsystem and data level dependent. The threshold can be needed not to loose the performance by frequent TClonesArray compression. I.e. for example it could look like in PndEmcMakeCluster::Exec() fClusterArray->Delete(); if (fClusterArray->Capacity()>100) fClusterArray->Compress(); However in case of EMC FairLinks created most problems at the simulation stage and here I do not know precisely where to compress TClonesArray. In PndEmc there is a method Reset() and I can guess it clear the memory at each event, but I am not 100% sure. void PndEmc::Reset() { fEmcCollection->Clear(); fPosIndex = 0; } So it could be modified like void PndEmc::Reset() { fEmcCollection->Clear(); if (fEmcCollection->Capacity()>100) fEmcCollection->Compress(); fPosIndex = 0; } Here I put for check of capacity 100 as an arbitrary number which should be optimized. I have complication to check it by myself since search for memory leaks requires rather powerful computer and mine is not. May be this will not help to solve the problem with FairLink but it can be checked. Dima
creator: Dima Melnychuk
date: 2010-11-03T12:45:51-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Dear Dima, I think this is a good idea. I will try to test it. You can only see the problem if you look at the memory consumption of pandaRoot during runtime. The problems are no memory leaks. Cheers, Tobias
creator: Tobias Stockmanns
date: 2010-11-03T13:17:30-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Dear pandaRooters, I have activated the FairLinks in the code again and replaced all the TClonesArray::Clear() with TClonesArray::Delete(). Please test if you encounter any problems or losses in performance. Cheers, Tobias
creator: Tobias Stockmanns
date: 2010-12-08T13:03:18-00:00

Re: Out of memory problem in EmcPoint - FairLink ?
type: item
description: Hi, it seems the fix solves the problem for EMC. Unfortunately it does not solve all the other crashes connected to TPC digitization and so on.
creator: StefanoSpataro
date: 2010-12-20T16:15:30-00:00