GSI Forum
GSI Helmholtzzentrum für Schwerionenforschung

Home » PANDA » PANDA - Computing » Grid and Infrastructure » spliting the job via collection
spliting the job via collection [message #12126] Thu, 23 June 2011 10:40 Go to next message
donghee is currently offline  donghee
Messages: 385
Registered: January 2009
Location: Germnay
first-grade participant
From: 195.37.186*
Hi all,

I have a technical problem.
Only 60 sub-jobs are allowed to me at Gridka.

But in my collection I have 200 files.
question # 1. how can I get only 60 file lists with find commend?

question # 2. How can I remove 140 files from existing collection list of total 200 files?

The problem is arising when I use split="file" or split="directory"
because I cannot control only until 60 sub-jobs.

If you have other simple method to avoid limiting of subjobs.
welcome to hear for that.

Thank you in advance.
Best wishes,
Donghee





Re: spliting the job via collection [message #12127 is a reply to message #12126] Thu, 23 June 2011 11:03 Go to previous messageGo to next message
donghee is currently offline  donghee
Messages: 385
Registered: January 2009
Location: Germnay
first-grade participant
From: 195.37.186*
Hi gridka veterans....

The collection list have 3 different file names with points, param, and pid.root in one particular events.

If I submit job via split="file" with a collection list,
Gridka submit count 3 files as each individual sub-jobs.

I think one has to submit with split="directory", then machine knows 3 files have to be puted in one subjob.
Is it correct? or it doesn't matter use either "file" or "directory"?

This is slightly confuse to me...(actually really complicate!)
Best wishes,
Dongee



Re: spliting the job via collection [message #12128 is a reply to message #12127] Thu, 23 June 2011 11:50 Go to previous messageGo to next message
donghee is currently offline  donghee
Messages: 385
Registered: January 2009
Location: Germnay
first-grade participant
From: 195.37.186*
Hi all,

I found simple methods

find -c output-run701 /panda/user/p/pbarprod/tdr11/output/sig/run701/*1/ evt_p*.root

Then only every *1 list will be accounted

and other solution is rather simple
One can add and remove items in the produced list with
addFileToCollection
removeFileFromCollection

But I'm still unclear for split="file" or "directory"


Cheers,
Donghee

Re: spliting the job via collection [message #12129 is a reply to message #12128] Thu, 23 June 2011 11:59 Go to previous message
StefanoSpataro is currently offline  StefanoSpataro
Messages: 2736
Registered: June 2005
Location: Torino
first-grade participant

From: *to.infn.it
"file" will create a subjob for each file in the collection, while "directory" for each folder. The "directory" solution is what you need, if you do not want to run the same code 3 times.
Previous Topic: test of submit script at gridka
Next Topic: check of submit job
Goto Forum:
  


Current Time: Sun Aug 01 13:35:37 CEST 2021

Total time taken to generate the page: 0.01913 seconds