Plan to test the following job and make the procedure fully automatic
(today 'cmsmon' machine has some trouble, so not able to work)

  • merge the following new files in /cms/mon/data/dropbox (one merge file per run, i.e. always two input files go into one merged file)
-rw-r--r--  1 dqmpro  dqmpro  7409853 Apr 29 18:23 DQM_DT_R000042847.root
-rw-r--r--  1 dqmpro  dqmpro  3483292 Apr 29 18:23 DQM_EcalBarrel_R000042847.root
-rw-r--r--  1 dqmpro  dqmpro  6844907 Apr 29 19:25 DQM_DT_R000042854.root
-rw-r--r--  1 dqmpro  dqmpro  3497098 Apr 29 19:25 DQM_EcalBarrel_R000042854.root
-rw-r--r--  1 dqmpro  dqmpro  4398548 Apr 29 19:42 DQM_DT_R000042859.root
-rw-r--r--  1 dqmpro  dqmpro  3481767 Apr 29 19:43 DQM_EcalBarrel_R000042859.root
-rw-r--r--  1 dqmpro  dqmpro  6728186 Apr 29 20:22 DQM_DT_R000042861.root
-rw-r--r--  1 dqmpro  dqmpro  4244196 Apr 29 20:22 DQM_EcalBarrel_R000042861.root
-rw-r--r--  1 dqmpro  dqmpro  5550923 Apr 29 20:34 DQM_DT_R000042862.root
-rw-r--r--  1 dqmpro  dqmpro  4235324 Apr 29 20:35 DQM_EcalBarrel_R000042862.root
-rw-r--r--  1 dqmpro  dqmpro  6015963 Apr 29 20:55 DQM_DT_R000042863.root
-rw-r--r--  1 dqmpro  dqmpro  4029006 Apr 29 20:55 DQM_EcalBarrel_R000042863.root
  • copy the merged files to the correct run directories in /cms/mon/data/dqm/results/000/... and so on.
  • make a backup copy of dqm.db that is presently at srv-c2d05-19:/home/dqm/dqm.db
    register the new files in dqm.db
    copy the new dqm.db to srv-c2d05-19:/home/dqm/dqm.db


Two things to do
  • Contact Bill Badgett about next steps concerning file storage on cmsmon
  • Write a script that transfers files from the local disks at srv-c2d05-1x to cmsmon without using nfs. Maybe the command rfcp file cmsmon:/cms/mon/data/dropbox will work.
    • But rfio is not intalled at srv-c2d05-1x
    • Is there any reason simple scp won't do the job?
    • Or in fact, just cp if the file system is mounted anyway? Or is it just mounted for reads?
    • At any rate, we should ask Bill for whatever protocol he prefers... Lassi can help anything get to work reliably, including "HTTP PUT".


Modified things in file merging script
  • the merge script look for new files in /cms/mon/data/.dropbox_test
  • The merge script always only merge files within one run.
  • The merged output file name is DQM_V*x_R*x_R*x.root
  • This merged file is created in /cms/mon/data/.dropbox_test and then moved into /cms/mon/data/dropbox_test/000/0xx/xxx/
  • The db-update script then use all the files in /cms/mon/data/dropbox_test/000/0xx/xxx/


  • file merging script should read data from /cms/mon/data/dqm/results/000/0xx/xxx
  • and put the merged file to /cms/mon/data/dqm/results/000/0xx/xxx


Problems in working in cmsmon account
  • scramv1 p CMSSW CMSSW_2_0_0_pre9 doesn't work in node, srv-c2d05-17
[cmsmon@srv-C2D05-17 dqm]$ scramv1 p CMSSW CMSSW_2_0_0_pre9
Cannot open "/cms/mon/.SCRAM/Environment" file for reading.
No such file or directory
  • need cmsmon password for checking out code from CVS
    [cmsmon@srv-c2c06-02 src]$ cvs -Q co DQMServices VisMonitoring?/DQMServer
    cmsmon@cmscvs.cern.ch's password:

Discussed things with Andreas

  • put script etc to /cms/mon/data/dqm/ (in cmsmon account)
  • work in cmsmon account : type sudo -u cmsmon -H csh in hkseo@cmsmon
  • watchdog: run crontab on cmsmon
  • merge always only for one run
  • register only merged files to db


  • wrote script to check file-merging process for cronjob


  • DB update problem was solved by replacing
    visDQMRegisterFile dbfile "Data taking" "Global run" rootfiles
    by visDQMRegisterFile dbfile "/Global/Online/ALL" "Global run" rootfiles
  • http://cmsmon.cern.ch/dqm/online not accessible suddenly
    • sudo -u cmsmon -H csh
    • ssh -ND 11080 cmsusr1
    • then set Socks Host: local host,  Port: 11080, proxy: in web broswer
    • search 'socks cms site:cern.ch' in google might be helpful to set it


modification of file-merging script


Slight modification of the file-merging script to do file registration once at the end of new file loop.


In order to run 'visDQMRegisterFile' in lxplus, tried to install DQM Gui server in lxplus but disk quota exceeded. Probably, I cannot use my home directory as DQM Gui server.


  • Implemented file registration to file-merging script.
  • How to update DB was given by Lassi (HowToUpdateDB)


Commited my file-merging script to CVS (DQM/Integration/scripts/)
  • cvs add mergeAndRegister_online.py
  • cvs commit mergeAndRegister_online.py

Next step is file registration (refer to VisMonitoring/DQMServer/scripts/visDQMRegisterFile)

  • creates a temporary copy of the master sqlite db
  • puts the information about the contents of the new file into the temporary copy of the DB
  • checks correctness of new DB and new merged file (!)
  • moves the modified DB into the place of the master DB


Made file-merging script with Python. This script will be in operation on 17.April.


Lassi summarized a proposal for naming DQM files and structuring the DQM data there-in discussed yesterday.

---- From Lassi ----

Here is a proposal for naming DQM files and structuring the DQM
data there-in, based on our discussions today.  This is an evolution
of the online convention described at:


This proposal accounts for the following four uses of DQM data:

   A) Online DQM for real data, live and archived data.
   B) Offline DQM for real data at Tier-0, CAF and Tier-1s.
   C) Offline DQM for monte-carlo data for CSA08.
   D) Offline DQM for monte-carlo data for release validation.

The proposal uses the following terms.  A $X designates a variable,
such as the actual run number.

   1. Stream and workflows.  The latter refers to the specific
      concept used in the production system (ProdAgent etc):
      The workflow changes whenever just about anything in the
      job configuration changes.  There are limited number of
      streams, O(50) in release validation (frequently referred
      to as a "dataset" there, not to be confused with the DBS
      dataset), O(3) in global runs, O(2) in online.

   2. Run and optional lumi section number.

   3. Producer.  The group producing the DQM data.  I call here
      the official CMS production the "ProdSys".  Online has two
      producers, "DQM" and "HLTDQM".  Each AlCa group on CAF is
      one producer.

   4. Step.  A stage in the processing chain.  Tier-0 has at
      least one step "PromptReco", and optionally a second one,
      the "1st-pass" reconstruction before copy to CAF, and each
      AlCa group would have the steps as shown in Rainer's schema.

   5. Subsystem.  The major group of DQM data, a detector group,
      or some physics classification for high-level monitoring.

The file naming and DQM structuring conventions are as follows.

   A) File structure as described in the DQMArchival twiki:
        Run $runnr/$subsystem/Run summary/$monitor_element_name

      File name structure:

      For example:
         "Run 38415/Hcal/Run summary/EventInfo/errorSummary" in:

   B) File structure as in A, with subsystems from DQMOffline.
      File name structure:
      For example DQM data for /Global-A/Online-CMSSW_2_0_0/RECO, run 

      Tier-1s follow same structure, they just have reprocessing
      workflow names instead of the prompt reconstruction names.

   C) File structure and file naming as in B, with:
       - Stream = primary datasets in the MC samples, about
         roughly the 28 request names on Oliver's slide 4.
       - Workflow = the derived full workflow names, expect
         up to factor of 3 more than actual datasets for
         restarts etc.
       - Run = artifical single forced number, i.e. each
         sample is considered a single run.

      *IMPORTANT NOTE* Forcing a single run number on each sample
      means the Tier-0 prompt reco DQM result becomes available only
      after the entire sample has been processed in its entirety!

   D) File structure:
      File name structure

      For example:
        "RelValQCD_Pt_80_120/GlobalHitsV/MCGeant/hMCG4Vtx1" in:
          RelVal/RelValQCD_Pt_80_120_CMSSW_2_0_0_V1.root (one relval)
          RelVal/RelVal_CMSSW2_0_0_V5.root (all release relvals merged)

-- HyunkwanSeo - 02 Apr 2008

Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2008-04-30 - HyunkwanSeo
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding KoreaCmsWiki? Send feedback