Long term plan
  • When files are moved from dropbox (in cmsmon) to castor, we should know how to update db file. Probably need to study 'sqlite3'
  • merge huge amount of files (~thousands of files) under results/000/xxx/ into one file

scheme for merge and register

worker node      
19 Gui <-> /home/dqm/dqm.db (production) dqm.db
18 testing <-> /home/dqm/dqm.db   dqm.db
17 /home/dqmpro/output/ cronjob cmsmon:~dropbox
16 /home/dqmpro/output/ cronjob cmsmon:~dropbox
15 /home/dqmpro/output/ cronjob cmsmon:~dropbox
  • run filesave_online.py on srv-c2d05-1[5-8]
  • 15,16,17 : copy DQM files to real dropbox, results, dqm.db->19
  • 18 : copy DQM files to dropbox_test, results_test, dqm.db->18


  • Found visDQMMergeFile doesn't work in cronjob because it need 'source setupForFileReg.csh' before execute it (in python, it cannot be implemented).
  • So watchdog script for 'merge script' should be made with shell script at current stage.
  • But still wondering why watchdog for 'A script' stopped once per hour???


Test of cronjob for 'A script' and 'merge script'
  • File copy by scp doesn't work in 'merge script' when it is started by cronjob (strange!!!)
  • 'A script' and 'merge script', when they are started by cronjob, stop about one time per hour


  • The tempral directory for 'A script' and 'merge script' might cause some conflict because they use the same directory (/cms/mon/data/.dropbox_tmp)


  • 'hadd' in lxplus and cmsmon use same version
      [lxplus215] ~/DQM/CMSSW_2_0_6/src > which hadd

      [hkseo@srv-c2c06-02 src]$ which hadd


  • replaced 'hadd' by 'visDQMMergeFile'
  • arranged the directories and files to separate test area and actual area completely.
  • the difference of the speed of 'hadd' between lxplus and cmsmon is due to root version??
  • should make 'A script' mentioned in https://twiki.cern.ch/twiki/bin/view/CMS/DQMArchival


test of the speed of merging process
  • merging following 6 files was tested.
-rw-r--r--  1 hkseo zh   40087 May 13 14:56 DQM_CSC_R000043640.root
-rw-r--r--  1 hkseo zh 1211964 May 13 14:57 DQM_DT_R000043640.root
-rw-r--r--  1 hkseo zh 3090072 May 13 14:57 DQM_EcalBarrel_R000043640.root
-rw-r--r--  1 hkseo zh   69503 May 13 14:57 DQM_Hcal_R000043640.root
-rw-r--r--  1 hkseo zh 5329888 May 13 14:57 DQM_L1T_R000043640.root
-rw-r--r--  1 hkseo zh   75560 May 13 16:50 DQM_RPC_R000043640.root

  • In lxplus
    • 'hadd' took 14 sec and 'visDQMMergeFile' took 30 sec.
  • In cmsmon with my account
    • 'hadd' took 55 sec. ('visDQMMergeFile' couldn't be installed in my account.)
  • In cmsmon with cmsmon account
    • 'hadd' took 60 sec and 'visDQMMergeFile' took 28 sec.

It seems like that 'hadd' is much slower in cmsmon, though 'visDQMMergeFile' doesn't make so different between lxplus and cmsmon.

How to merge file without using 'hadd' (provided by Lassi)

Check out the head of VisMonitoring/DQMServer, build it, update 
environment, and make sure $PATH has been updated:

   scram p CMSSW CMSSW_2_0_6
   cd CMSSW_2_0_6/src
   cvs -Q co VisMonitoring/DQMServer
   scram b -j 10
   eval `scram runtime -sh`

Then merge using e.g.:

$ time visDQMMergeFile foo.root /cms/mon/data/dqm/results/000/043/390/
Timing information: 25.67s real, 12.58s user, 1.00s system (52%%)

I have *NOT* tested it doesn't lose monitoring elements (or won't 
invent new ones), although the output file is approximately the right 



Things to do :
set up the merge-system without the mount from srv-c2d05-19.

  • setup and run the merge script on each of the new runs in the dropbox
  • copy the merged files into the run directory
  • run the register script to create a new dqm.db (that replaces the one at srv-c2d05-19:/home/dqm)
  • setup the copy script for files from srv-c2d05-19 using scp

-- HyunkwanSeo - 02 May 2008

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r10 - 2008-06-02 - HyunkwanSeo
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding KoreaCmsWiki? Send feedback