2008/04/30

Plan to test the following job and make the procedure fully automatic
(today 'cmsmon' machine has some trouble, so not able to work)

  • merge the following new files in /cms/mon/data/dropbox (one merge file per run, i.e. always two input files go into one merged file)
-rw-r--r--  1 dqmpro  dqmpro  7409853 Apr 29 18:23 DQM_DT_R000042847.root
-rw-r--r--  1 dqmpro  dqmpro  3483292 Apr 29 18:23 DQM_EcalBarrel_R000042847.root
-rw-r--r--  1 dqmpro  dqmpro  6844907 Apr 29 19:25 DQM_DT_R000042854.root
-rw-r--r--  1 dqmpro  dqmpro  3497098 Apr 29 19:25 DQM_EcalBarrel_R000042854.root
-rw-r--r--  1 dqmpro  dqmpro  4398548 Apr 29 19:42 DQM_DT_R000042859.root
-rw-r--r--  1 dqmpro  dqmpro  3481767 Apr 29 19:43 DQM_EcalBarrel_R000042859.root
-rw-r--r--  1 dqmpro  dqmpro  6728186 Apr 29 20:22 DQM_DT_R000042861.root
-rw-r--r--  1 dqmpro  dqmpro  4244196 Apr 29 20:22 DQM_EcalBarrel_R000042861.root
-rw-r--r--  1 dqmpro  dqmpro  5550923 Apr 29 20:34 DQM_DT_R000042862.root
-rw-r--r--  1 dqmpro  dqmpro  4235324 Apr 29 20:35 DQM_EcalBarrel_R000042862.root
-rw-r--r--  1 dqmpro  dqmpro  6015963 Apr 29 20:55 DQM_DT_R000042863.root
-rw-r--r--  1 dqmpro  dqmpro  4029006 Apr 29 20:55 DQM_EcalBarrel_R000042863.root
  • copy the merged files to the correct run directories in /cms/mon/data/dqm/results/000/... and so on.
  • make a backup copy of dqm.db that is presently at srv-c2d05-19:/home/dqm/dqm.db
    register the new files in dqm.db
    copy the new dqm.db to srv-c2d05-19:/home/dqm/dqm.db

2008/04/29

Two things to do
  • Contact Bill Badgett about next steps concerning file storage on cmsmon
  • Write a script that transfers files from the local disks at srv-c2d05-1x to cmsmon without using nfs. Maybe the command rfcp file cmsmon:/cms/mon/data/dropbox will work.
    • But rfio is not intalled at srv-c2d05-1x
    • Is there any reason simple scp won't do the job?
    • Or in fact, just cp if the file system is mounted anyway? Or is it just mounted for reads?
    • At any rate, we should ask Bill for whatever protocol he prefers... Lassi can help anything get to work reliably, including "HTTP PUT".

2008/04/26

Modified things in file merging script
  • the merge script look for new files in /cms/mon/data/.dropbox_test
  • The merge script always only merge files within one run.
  • The merged output file name is DQM_V*x_R*x_R*x.root
  • This merged file is created in /cms/mon/data/.dropbox_test and then moved into /cms/mon/data/dropbox_test/000/0xx/xxx/
  • The db-update script then use all the files in /cms/mon/data/dropbox_test/000/0xx/xxx/

2008/04/25

  • file merging script should read data from /cms/mon/data/dqm/results/000/0xx/xxx
  • and put the merged file to /cms/mon/data/dqm/results/000/0xx/xxx

2008/04/23

Problems in working in cmsmon account
  • scramv1 p CMSSW CMSSW_2_0_0_pre9 doesn't work in node, srv-c2d05-17
[cmsmon@srv-C2D05-17 dqm]$ scramv1 p CMSSW CMSSW_2_0_0_pre9
Cannot open "/cms/mon/.SCRAM/Environment" file for reading.
No such file or directory
  • need cmsmon password for checking out code from CVS
    [cmsmon@srv-c2c06-02 src]$ cvs -Q co DQMServices VisMonitoring?/DQMServer
    cmsmon@cmscvs.cern.ch's password:

Discussed things with Andreas

  • put script etc to /cms/mon/data/dqm/ (in cmsmon account)
  • work in cmsmon account : type sudo -u cmsmon -H csh in hkseo@cmsmon
  • watchdog: run crontab on cmsmon
  • merge always only for one run
  • register only merged files to db

2008/04/21

  • wrote script to check file-merging process for cronjob

2008/04/18

  • DB update problem was solved by replacing
    visDQMRegisterFile dbfile "Data taking" "Global run" rootfiles
    by visDQMRegisterFile dbfile "/Global/Online/ALL" "Global run" rootfiles
  • http://cmsmon.cern.ch/dqm/online not accessible suddenly
    • sudo -u cmsmon -H csh
    • ssh -ND 11080 cmsusr1
    • then set Socks Host: local host,  Port: 11080, proxy: 127.0.0.1 in web broswer
    • search 'socks cms site:cern.ch' in google might be helpful to set it

2008/04/17

modification of file-merging script

2008/04/15

Slight modification of the file-merging script to do file registration once at the end of new file loop.

2008/04/10

In order to run 'visDQMRegisterFile' in lxplus, tried to install DQM Gui server in lxplus but disk quota exceeded. Probably, I cannot use my home directory as DQM Gui server.

2008/04/09

  • Implemented file registration to file-merging script.
  • How to update DB was given by Lassi (HowToUpdateDB)

2008/04/08

Commited my file-merging script to CVS (DQM/Integration/scripts/)
  • cvs add mergeAndRegister_online.py
  • cvs commit mergeAndRegister_online.py

Next step is file registration (refer to VisMonitoring/DQMServer/scripts/visDQMRegisterFile)

  • creates a temporary copy of the master sqlite db
  • puts the information about the contents of the new file into the temporary copy of the DB
  • checks correctness of new DB and new merged file (!)
  • moves the modified DB into the place of the master DB

2008/04/05

Made file-merging script with Python. This script will be in operation on 17.April.

2008/04/02

Lassi summarized a proposal for naming DQM files and structuring the DQM data there-in discussed yesterday.

---- From Lassi ----
Hi,

Here is a proposal for naming DQM files and structuring the DQM
data there-in, based on our discussions today.  This is an evolution
of the online convention described at:

   https://twiki.cern.ch/twiki/bin/view/CMS/DQMArchival

This proposal accounts for the following four uses of DQM data:

   A) Online DQM for real data, live and archived data.
   B) Offline DQM for real data at Tier-0, CAF and Tier-1s.
   C) Offline DQM for monte-carlo data for CSA08.
   D) Offline DQM for monte-carlo data for release validation.

The proposal uses the following terms.  A $X designates a variable,
such as the actual run number.

   1. Stream and workflows.  The latter refers to the specific
      concept used in the production system (ProdAgent etc):
        /$STREAM/$VERSION-ETC-INFO/$TIER
      The workflow changes whenever just about anything in the
      job configuration changes.  There are limited number of
      streams, O(50) in release validation (frequently referred
      to as a "dataset" there, not to be confused with the DBS
      dataset), O(3) in global runs, O(2) in online.

   2. Run and optional lumi section number.

   3. Producer.  The group producing the DQM data.  I call here
      the official CMS production the "ProdSys".  Online has two
      producers, "DQM" and "HLTDQM".  Each AlCa group on CAF is
      one producer.

   4. Step.  A stage in the processing chain.  Tier-0 has at
      least one step "PromptReco", and optionally a second one,
      the "1st-pass" reconstruction before copy to CAF, and each
      AlCa group would have the steps as shown in Rainer's schema.

   5. Subsystem.  The major group of DQM data, a detector group,
      or some physics classification for high-level monitoring.

The file naming and DQM structuring conventions are as follows.

   A) File structure as described in the DQMArchival twiki:
        Run $runnr/$subsystem/Run summary/$monitor_element_name

      File name structure:
        $runnr[1:3]/$runnr[4:6]/$runnr[7:9]/
          $producer_$subsystem_R$runnr_V$version.root
          $producer_$subsystem_RR$firstrun_$lastrun_V$version.root
          $producer_$subsystem_R$runnr_L$luminr.root
          $producer_$subsystem_R$runnr_E$eventnr.root
          $producer_$subsystem_R$runnr_T$utctime.root

      For example:
         "Run 38415/Hcal/Run summary/EventInfo/errorSummary" in:
           000/038/415/DQM_Hcal_R000038415_V1.root
           000/038/415/HLTDQM_EcalBarrel_R000038415_V1.root
           000/038/415/DQM_Merged_R000038415_V3.root

   B) File structure as in A, with subsystems from DQMOffline.
      File name structure:
        $producer/$step/$stream/$runnr[1:3]/$runnr[4:6]/$runnr[7:9]/
          DQM_R$runnr_V$version__$workflow.root
      For example DQM data for /Global-A/Online-CMSSW_2_0_0/RECO, run 
38415:
        ProdSys/PromptReco/Global-A/000/038/415/
DQM_R0000038415_V1__Global-A__Online-CMSSW_2_0_0__RECO.root
        ProdSys/1st-pass/Global-A/000/038/415/
DQM_R0000038415_V1__Global-A__Online-CMSSW_2_0_0__RECO.root
        Tracker/CAF1/Global-A/000/038/415/DQM_R0000038415_V1__Global-
A__Online-CMSSW_2_0_0__RECO.root

      Tier-1s follow same structure, they just have reprocessing
      workflow names instead of the prompt reconstruction names.

   C) File structure and file naming as in B, with:
       - Stream = primary datasets in the MC samples, about
         roughly the 28 request names on Oliver's slide 4.
       - Workflow = the derived full workflow names, expect
         up to factor of 3 more than actual datasets for
         restarts etc.
       - Run = artifical single forced number, i.e. each
         sample is considered a single run.

      *IMPORTANT NOTE* Forcing a single run number on each sample
      means the Tier-0 prompt reco DQM result becomes available only
      after the entire sample has been processed in its entirety!

   D) File structure:
       $stream/$subsystem/$monitor_element_name
      File name structure
       $producer/$stream_$release_V$version.root

      For example:
        "RelValQCD_Pt_80_120/GlobalHitsV/MCGeant/hMCG4Vtx1" in:
          RelVal/RelValQCD_Pt_80_120_CMSSW_2_0_0_V1.root (one relval)
          RelVal/RelVal_CMSSW2_0_0_V5.root (all release relvals merged)

-- HyunkwanSeo - 02 Apr 2008

Edit | Attach | Watch | Print version | History: r16 < r15 < r14 < r13 < r12 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r16 - 2008-04-30 - HyunkwanSeo
 
KOREA-CMS
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding KoreaCmsWiki? Send feedback