his document is a repository of common problems (with solutions) pertaining to the configuration and operation of the ALICE LCG VO-Box. For details on the VO-Box description and installation, please see the LCGVO-Box_HowTo and the ALICE-specific installation guide. For xrootd-over-DPM installation and debugging, please check the relevant HowTo^?.

 

 

General Notes

Please remember that the proxy you use to log in on the VO-Box via gsissh (the login proxy) is not the same that is used to start AliEn services and submit jobs to the LCG (the user proxy; please check HowToManageVOBoxProxies for definitions and many more details about proxy management on the VO-Box).

In order to run AliEn with your personal login proxy, do

 

and to use the registered renewable user proxy used for services do

 

A very useful option to AliEn is --debug 10. Please see the AliEn User Guide for more information about AliEn itself.

 

How to find LCG, AliEn and JobAgent job IDs

Coming soon...

 

Where to find log files, configurations & such

 

• All the AliEn services log files (ClusterMonitor.log, CE.log, SE.log, PackMan.log, MonaLisa.log), plus some other interesting stuff are on the VO-Box, in the ~alicesgm/alien-logs/ directory.

• The JobAgent execution logs are in ~alicesgm/alien-logs/proc/<JobId>, where <JobId> is the AliEn JobId (not the LCG one, nor the JobAgent ID).

• The DB with the correspondence between JobAgentId, LCG JobID and (when available) Alien JobId, plus some more information about currently running and queued JobAgents, is in ~alicesgm/alien-logs/CE.db/JOBAGENT; another file in that directory, ~alicesgm/alien-logs/CE.db/JOBIDS holds a list of all LCG JobIds ever submitted through the VO-Box.

• The local configuration files are ~alicesgm/.alien/Environment (that is sourced just before the execution of any AliEn command) and, when relevant, ~alicesgm/.alien/alice.conf. See the VO-Box installation guide guide for details about those two files.

• The central configuration repository is ldap://aliendb5.cern.ch:8389/o=alice,dc=cern,dc=ch; you can use ldapsearch or any LDAP browser (like GQ) to browse the entries.

 

Downloading the stdout/stderr of a JobAgent

To download the stdout/stderr of a given JobAgent, you first need to locate its LCG unique JobId, which is an HTTPS URL with the RB address and a GUID, e.g. https://gridit-rb-01.cnaf.infn.it:9000/ukVYsNY0m5IlN4cdtS88BA. This is printed out on the CE logfile ~alicesgm/alien-logs/CE.log when the job is first submitted, and on the Cluster Monitor log in a couple of other occurrences, for example

 

It is also saved in ~alicesgm/alien-logs/CE.db/JOBAGENT and ~alicesgm/alien-logs/CE.db/JOBIDS (see above). You can download the output of the JobAgent by issuing

 

Example:

 

The jobs are owned by the owner of the user proxy that was used to submit them. If the subject of such user proxy is not the same as yours (i.e. the same of your login proxy) you will not have the privilege to download those job's output. Try to guess which one was used (very likely to be one of those in /opt/vobox/alice/proxy-repository/) and use that one:

 

where <proxy file> will be something like /opt/vobox/alice/proxy_repository/+2fC+3dCH+2fO+3dCERN+2fOU+3dGRID+2fCN+3dPatricia+20Mendez+20Lorenzo-ALICE+2fCN+3dproxy.

 

Restarting the services

To restart all of the services at once, use the aliend startup script:

 

To start a single service, use the script that sets the right user proxy:

 

where can be any of CE, SE, Monitor, MonaLisa, PackMan.

Whenever restarting the services at any site, please check that the ALIEN_USER set in the ~alicesgm/.alien/Environment file matches one of the proxies registered in the VO-Box proxy DB. You can check them by issuing

 

Example:

 

You can find the certificate subject corresponding to a given AliEn user name by issuing

 

The funny perl command in the end is needed to take care of the nasty habit of ldapsearch of splitting long lines.

If none of the proxies matches the ALIEN_USER, simplest thing is to change it to your AliEn user name and register another proxy for yourself. For more details about proxy management on the VO-Box, please see the HowToManageVOBoxProxies document.

Important: If you change the ALIEN_USER in ~alicesgm/.alien/Environment, it is mandatory to restart all the services, in order to have them running with the appropriate credentials.

 

Useful AliEn commands

Here's a synopsis of some AliEn commands that can be useful while debugging the system:

 

• top  -id <jobid>
• top [-status <status>|-all_status] [-host <host>]
• registerOutput <jobId>
• ps [options]
• ps jdl <jobId>
• ps trace [all|trace|...] <jobId>
• spy
• queueinfo [-jdl] [<queuename>]

 

Useful LCG commands

Here's a synopsis of some LCG/gLite commands that can be useful while debugging the system:

 

• lcg-infosites --vo alice [ce|se|rb|...]
• lcg-version

 

---

 

Problem: Cannot start the services

Symptoms: You cannot start any service: AliEn dies with messages similar to the following:

 

Diagnosis: This is probably a problem with the central services; a possible cause is a crash of the Authen server, but there are others. You can check the status of the central services by using the central services page of the ALICE MonALISA repository.

 

---

Symptoms: You cannot start any service. The aliend startup script complains about proxies

 

Diagnosis: There is no valid proxy registered in the VO-Box database that matches the ALIEN_USER set in ~alicesgm/.alien/Environment. Check it by issuing the following commands:

 

Solution: If there is no proxy registered (i.e. the above command returns nothing) register one and check that the ALIEN_USER in ~alicesgm/.alien/Environment matches the owner of the proxy. Alternatively, if there is a valid proxy, change ALIEN_USER to the owner of that proxy.

 

In some cases, the proxy DB on the VO-Box (/opt/vobox/alice/_registerer_proxies.db) can become corrupt. Evidence for this is usually that the list of proxies returned by vobox-proxy --vo alice --dn all query does not match the files in /opt/vobox/alice/proxy_repository, i.e. there are some extra files in the repository.

Solution: Clean up the repository and register again a fresh proxy:

 

 

Please check also HowToManageVOBoxProxies for (many) more details on proxy management for the VO-Box.

 

---

Symptoms: AliEn services do not start at boot time. There are messages in the boot log that complain about proxies:

 

Diagnosis: The scripts that starts VO-Box services (/opt/vobox/alice/agents/alice-box-proxyrenewal) goes through the startup script in /opt/vobox/alice/start/ /before/ starting the Proxy Renewal Service. So, if while the service or the machine were down, the proxy expired the services cannot start.

Solution: The startup script will likely be changed; in the meanwhile, the services need to be started by hand after the boot sequence:

 

 

---

 

Problem: Services up but no jobs running

Symptoms: All the services are running on the VO-Box, but it does not submit any JobAgent, i.e. no ALICE jobs seen by the LRMS.

Diagnosis: One possibility is a problem with the CE GRIS (or BDII). This can be checked by looking at the CE log ~alicesgm/alien-logs/CE.log, where periodically a message like this can be found:

 

If all the numbers (with the possible exception of JA:) are 0, then it's highly likely that the VO-Box can't get these numbers from the CE GRIS. Another possibility is having all zeroes, except W:4444. This is a problem with the site LRMS.

Solution: Check with the LCG site admins to check the CE GRIS.

You can find out the name and status of CEs, SEs or other LCG services using the lcg-infosites command. For example (please try lcg-infosites --help for much more options):

 

 

---

Symptoms: The system submits no JobAgent to the RB, and there are messages like "No job matches your JDL" in the ~alicesgm/alien-logs/CE.log. Furthermore, there's no package listing in the head of the same file.

Diagnosis: The PackMan is dead or somehow stuck. This can happen e.g. if the CE is restarted with a different proxy. In normal conditions, the site JDL which is printed out at the beginning of the CE logfile contains package info from the PackMan:

 

The full site JDL is also printed out when you issue alien login. You can also check that PackMan is working by issuing (from the AliEn command line) packman list or packman listInstalled.

Solution: Restart the PackMan:

 

 

---

Symptoms: The certificate files look OK, and the error in the ~alicesgm/alien-logs/CE.log is different:

 

The lines starting with 'Connecting to host' and 'Logging to host' show different host names (i.e. the Network Server and the Loggin&Bookkeeping hosts are different); the actual error messages may be different.

Diagnosis: There is very likely a problem with the RB. You can try using a different one (see below). On top of that, the system is trying to send logging information to a different default host, and this can mess up the things a bit.

Solution: The submission commands take their configuration (including the RB to use) from a file which by default is in /opt/edg/etc/alice/edg_wl_ui.conf. The file looks something like

 

Root privileges are needed to change the default file; however, it is possible to tell the system to use another file. Copy the relevant configuration file to some place where you can edit it, change it to point to another RB and instruct AliEn to pass it to edg-job-submit:

 

1. Copy /opt/edg/etc/alice/edg_wl_ui.conf to e.g. /home/alicesgm/.alien/edg-wl-ui.conf

2. Edit it to point to your favorite RB. Be sure to change both entries (NSAddresses and LBAddresses).

3. Add to the ~alicesgm/.alien/alice.conf a line like

 

1. Restart the CE: $VO_ALICE_SW_DIR/alien/scripts/lcg/lcgAlien.sh StartCE

To check whether you're passing any option to edg-job-submit, look in the CE log ~alicesgm/alien-logs/CE.log for a line like

To fix the logging problem, the LoggingDestination entry in the /opt/edg/etc/edg_wl_ui_cmd_var.conf file should be removed (this requires root access).

 

---

Symptoms: Move this discussion to the installation HowTo Start the CE with --debug 10 option and you may see something like "No match found" etc

 

Diagnosis: AliEn can't match user requirements to your site. Check the user's JDL file, as reported by ~alicesgm/alien-logs/CE.log, and "your CE jdl" printed in the beginning of CE.log. You can see, for example the LocalDiskSpace entry, which may be less that user's JDL wants.

Solution: Define in your ~/.alien/Environment

 

and put there some dir where there are few GB of space, to satisfy most of requirements.

There are actually two different environment files. The ~/.alien/Environment, which we usually use for configuration on VO-Boxes, is local to the VO-Box and takes precedence upon the $VO_ALICE_SW_DIR/alien/.Environment, which (since it resides on the shared partition) is visible also from Worker Nodes.

 

 

---

Symptoms: All the services are running on the VO-Box, JobAgents are submitted (i.e. the LCG site see many ALICE jobs) but they don't pick up any real jobs (i.e. they 're about 3 minutes long and use next to no CPU; from MonaLisa, you see no AliEn jobs running on that site).

Diagnosis: The most likely problem (provided all the correct ports between WNs, the VO-Box and the AliEn server are open) is that you forgot the -t 48 option while registering the myProxy, and the proxy renewal get short proxies. You can check this by downloading (with lcg-job-get-output, see above) the output of one of the JobAgents. The file should contain something like

 

if the second number is much less than the first, this is likely to be the problem.

From AliEn 2.10, an error message will be printed in the ~alicesgm/alien-logs/CE.log file if the renewed proxy is too short because of this problem.

Solution: Re-register the MyProxy. From any LCG VO-Box, issue

 

Then log in on the VO-Box and issue

 

answering 'yes' (twice) when asked whether you want to overwrite the existing proxy. No AliEn services need restarting.

 

---

 

Problem: Cannot start the Clustermonitor

Symptoms: In the Clustermonitor.log you can see an error like:

 

Diagnosis: The central IS does not recognize your site as an ALICE site

Solution: The domain of the site has to be added to the entry 'ou=LCG,ou=Sites,o=alice,dc=cern,dc=ch' in the LDAP server

 

---

 

 

Problem: Jobs finish with status ERROR_V

Symptoms: There are a lot of jobs in your site that finish with 'ERROR_V' (error validating).

Diagnosis: ERROR_V means that from the point of view of AliEn, the job ran without problems. However, the job has a validation procedure, and that validation procedure thought that there was something wrong. For instance, this error can happen if the job tries to access a library that it is not in the worker node.

Solution: The first thing to do is to check the output of any of those jobs. The jobs that finish with ERROR_V do not register the output in the catalogue. However, you can use the command registerOutput <jobid> to create the links and check what happened.