User How-To - Catalogue | JAliEn - ALICE Environment Grid Framework

Basic commands

If you got this far, you have a working installation of alien and are able to authenticate to the system. Congratulations!!

Now, let's take a look at the different commands that you can execute inside alien. The first thing that you have to do is type alien:

[pcepalice45] ~ > alien
Connecting to database alien_system in aliendb.cern.ch:3307 ...
[aliendb.cern.ch:3307] / >

From this prompt, you can pass any commands to the AliEn System. Be careful, because even if it looks like a UNIX shell, it IS NOT a UNIX shell.

To get the full list of the commands that you can use, press tab twice:

[aliendb.cern.ch:3307] / > <tab> <tab>

find update removeTagValue zoom ls mirror cat
register addTagValue user addMirror exit quit mkdir
addTag help showTags chgroup addUser whereis chmod
tree chown pwd passwd host deleteMirror echo
umask showMirror rmdir cd cp groups whoami
get showTagValue remove debug add addHost mkremdir

Most of the commands are similar to the standard UNIX commands:

[aliendb.cern.ch:3307] / > whoami
psaiz
[aliendb.cern.ch:3307] / > ls -al
drwxr-xr-x admin admin 0 Jul 17 18:08 .
drwxr-xr-x admin admin 0 Aug 07 15:03 bin
drwxr-xr-x admin admin 0 Apr 02 12:34 jdl
drwxr-xr-x admin admin 0 Sep 20 15:40 proc
drwxr-xr-x admin admin 0 Sep 07 10:10 alice [AlienOnlyGroup="z2"]
[aliendb.cern.ch:3307] / > cd alice

Current path is: /alice/
[aliendb.cern.ch:3307] /alice/ > ls
bin
cern.ch
jdl
packages
simulation
tags
[aliendb.cern.ch:3307] /alice/ > cd bin
Current path is: /alice/bin/
[aliendb.cern.ch:3307] /alice/bin/ > ls -al 
drwxr-xr-x admin admin 0 Nov 06 12:43 .
drwxr-xr-x admin admin 0 Nov 06 12:43 ..
-rwxr-xr-x admin admin 637 Nov 06 17:48 AliRoot.sh
-rwxr-xr-x admin admin 1987 Nov 08 20:22 AliRoot.sh.Validate
-rwxr-xr-x admin admin 64 Apr 11 16:10 Analysis.sh
-rwxr-xr-x admin admin 555 Nov 09 11:32 SubmitJobs.sh
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > help
Possible commands:
add -> Add file to the catalog
addTag -> Creates a new tag and asociates it with a directory
...
[aliendb.cern.ch:3307] / > echo

Configuration: AUTHEN_DATABASE = 'ADMIN'
Configuration: AUTHEN_DRIVER = 'mysql'
Configuration: AUTHEN_HOST = 'aliendb.cern.ch:3307'
Configuration: AUTH_HOST = 'alien.cern.ch'
Configuration: AUTH_PORT = '8080'
Configuration: CACHE_DIR = '/tmp/AliEn/cache'
...

Finally, to exit the catalogue, you can use 'exit', 'quit', or &lt Ctrl>+c

[aliendb.cern.ch:3307] /alice/bin/ > quit
bye now!

If you want to execute only one command, you can do it also from the UNIX prompt (not the alien prompt!), typing 'alien --exec &lt command >'

[pcepalice45] ~ > alien -exec ls -l /alice/
drwxr-xr-x admin admin 0 Nov 06 12:43 bin
drwxr-xr-x admin admin 0 Sep 07 10:15 cern.ch
drwxr-xr-x admin admin 0 Apr 03 14:52 jdl
drwxr-xr-x admin admin 0 Nov 27 12:27 packages
drwxr-xr-x aliprod z2 0 Oct 18 20:00 simulation
drwxr-xr-x admin admin 0 Sep 07 10:15 tags
[pcepalice45] ~ >

Getting files

All the entries that you can see in the catalogue are not real files, but Logical File Names (LFN). You can think of an LFN like an index that points to one

(or more) Physical File Names (PFN). The physical file name points to a copy of the file. It can be on a disk, tape, or any kind of mass storage system.

Examples of pfn's are:

castor://alienx.cern.ch/castor/cern.ch/user/a/aliprod/sim//2001-02/V3.06.02/00014/00056/galice.root
hsi://pdsflx002.nersc.gov/nersc/projects/alice/2001-02/V3.06.02/00034/00056/galice.root
tape://alienx.cern.ch/Y27671/52?size=274&bsize=26214

If an LFN points to more than one PFNs, all these pfns are just replicas (also known as mirrors). However, a normal user does not have to care about PFNs.

All (s)he should be concern about is the LFN. The user can get an entry from the catalogue, and the AliEn will bring the pfn that is closest to the user. To do

that, (s)he must type:

[aliendb.cern.ch:3307] /alice/bin/ > get
Error: not enough arguments in get
Usage: get [-n] &lt file> [&lt localfile>]
[aliendb.cern.ch:3307] /alice/bin/ > get AliRoot.sh

Apr 19 14:35 info Getting the file
soap://aliendb.cern.ch:8092/home/alienMaster/catalogue/alice/bin/AliRoot.sh?URI=SE
And the file is /tmp/AliEn/tmp/proc44269/command
[aliendb.cern.ch:3307] /alice/bin/ > get AliRoot.sh /home/psaiz/AliRoot.sh
Sep 10 16:18 info Getting the file
soap://aliendb.cern.ch:8092/home/alienMaster/catalogue/alice/bin/AliRoot.sh?URI=SE
And the file is /home/psaiz/AliRoot.sh
[aliendb.cern.ch:3307] /alice/bin/ >

And this will bring the file locally to the user (in the example, the file /tmp/AliEn/tmp/proc44269/command will be a copy of one of the pfn's pointed to by the

lfn /alice/bin/AliRoot).

If the user tries to get the same file again, the local copy will be returned (unless it was deleted!!)

You can also see the file, if you use the command cat

[aliendb.cern.ch:3307] /alice/bin/ > cat AliRoot.sh

#!/bin/sh

cd $HOME; ARGS=$*

while [ $# -gt 0 ]
do
case $1 in
--round)
shift 1
echo "ALIEN-COMMENT:ROUND=$1"
shift 1
;;
--event)
shift 1
echo "ALIEN-COMMENT:EVENT=$1"
shift 1
;;
--run)
shift 1
echo "ALIEN-COMMENT:RUN=$1"
shift 1
;;
--catalogpath)
shift 1
echo "ALIEN-COMMENT:PATH=$1"
shift 1
;;

*)
shift 1
;;
esac
done

alien -x $HOME/.alien/AliEn/scripts/AliRoot.pl $ARGS

However, if you do want to know where the file is stored, you have the command whereis, which will give you the PFN(s) and SE where a LFN is stored

[aliendb.cern.ch:3307] /alice/bin/ > whereis AliRoot.sh
And the file is in Alice::CERN::castor
soap://aliendb.cern.ch:8092/home/alienMaster/catalogue/alice/bin/AliRoot.sh?URI=SE
[aliendb.cern.ch:3307] /alice/bin/ > whereis /alice/simulation/2002-04/V3.08.Rev.01/00002/00073/galice.root
And the file is in
Alice::FZK::File file://c01-001-110/grid/fzk.de/alice/alice1/AliEn/save/2002-04/V3.08.Rev.01/00002/00073/galice.root
Alice::CERN::Castor castor://wacdr001d.cern.ch/castor/cern.ch/user/a/aliprod/AliEn-1.0/data//2002-04/V3.08.Rev.01/0
0002/00073/galice.root

Putting files in the catalogue

Every user has a home directory, where (s)he can register any files. There are two ways to register a file: If the file is already in a SE, or if it can be

accessed from anywhere, you can use the command register. This will create a link to that file. To do it, you will have to give the LFN and PFN

[aliendb.cern.ch:3307] /alice/bin/ > cd
Current path is: /alice/cern.ch/user/p/psaiz/
[aliendb.cern.ch:3307] /alice/bin/ > register

Error in register: not enough arguments
Usage register <path> <pfn> [ <size> [<storage element>]]
Pfn: castor://<host>/<path>
aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > register Tutorial.html http://alien.cern.ch/Doc/tutorial.html
File /alice/cern.ch/user/p/psaiz/Tutorial.html inserted in the catalog
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > cat Tutorial.html

&lt !DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
&lt html>
&lt head>
&lt title>AliEn Tutorial &lt /title>
...

If the file is in your laptop, or in a place where it will not be accesible from other places, AliEn will make a copy of it in one of the SE. By default, it will be

the SE's of your site, although you can specify another one:

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > add

Error: not enough parameters
Usage: add <lfn> <pfn> [<SE> [<previous storage element>]]
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > add alien file://pcepalice45.cern.ch/home/psaiz/bin/alien
Sep 10 16:32 info Registering the file file://pcepalice45.cern.ch/home/psaiz/bin/alien in
Sep 10 16:32 info Copying a file into an SE
Sep 10 16:32 info Getting the file file://pcepalice45.cern.ch/home/psaiz/AliEn/local//00001/00041 of size 5167
File /alice/cern.ch/user/p/psaiz/alien inserted in the catalog
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ >

Replicating files

You can use the AliEn catalogue to replicate files from one site to another. The LFN will point then to several PFN's, and depending on the location of

the client, AliEn will decide to pick the closest PFN.

The command to create this replicas is 'mirror'. You have to specify the LFNand the storage element where you want to replicate the file (if you don't

specify any storage element, AliEn will choose your closest storage element).

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > whereis alien
And the file is in Alice::CERN::filepsaiz file://pcepalice45.cern.ch/home/psaiz/AliEn/local//00001/00041
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > mirror alien Alice::Torino::File
Jun 11 16:10 info Mirroring file /alice/cern.ch/user/p/psaiz/alien
Jun 11 16:10 info Transer has been scheduled (transfer 28979)
Jun 11 16:10 info Waiting until the transfer is completed...
Jun 11 16:10 info Asking if the transfer has finished...
Jun 11 16:10 info Got INSERTING
Jun 11 16:11 info Asking if the transfer has finished...
Jun 11 16:11 info Got INSERTING
Jun 11 16:11 info Asking if the transfer has finished...
Jun 11 16:11 info Got WAITING
Jun 11 16:11 info Asking if the transfer has finished...
Jun 11 16:11 info Got LOCAL COPY
Jun 11 16:12 info Asking if the transfer has finished...
Jun 11 16:12 info Got LOCAL COPY
Jun 11 16:12 info Asking if the transfer has finished...
Jun 11 16:12 info Got CLEANING
Jun 11 16:12 info Asking if the transfer has finished...
Jun 11 16:12 info Got DONE
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > whereis alien
And the file is in Alice::CERN::filepsaiz file://pcepalice45.cern.ch/home/psaiz/AliEn/local//00001/00041
Alice::Torino::File file://alifarm01.to.infn.it/alinas/aliprod/00001/00052.1055340729

MetaData Catalogue

You can use the AliEn catalogue not only for mapping lfn's into pfn's, but also to include more data about your files. The extra information that you

assigned to the files is call Metadata, or tags. First of all, you have to describe what kind of metadata you want to store. This is done with the files in

the directory /&lt org >/tags. These files describe the columns of a table that will be created in the database. For instance, in the case of ALICE,

there is a tag called MonteCarloRuns.

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > cd /alice/tags/
Current path is: /alice/tags/
[aliendb.cern.ch:3307] /alice/tags/ > ls -al
drwxr-xr-x admin admin 0 Sep 07 10:15 .
drwxr-xr-x admin admin 0 Sep 07 10:15 ..
-rwxr-xr-x admin admin 247 May 24 15:58 MonteCarloRuns
-rwxr-xr-x admin admin 61 May 24 15:59 standard
[aliendb.cern.ch:3307] /alice/tags/ > cat MonteCarloRuns

eventGen char(15), GluonRadiation int(1), HolesPHOSRICH int(1), trigger int(1), ptminjet int(4), impactparametermin
float, impactparametermax float, thetamin float, thetamax float, phimin float, phimax float, multiplicity int(7),
radiation int(2)
[aliendb.cern.ch:3307] /alice/tags/ >

Once you have defined the tags, you assigned them to a directory.

[aliendb.cern.ch:3307] /alice/tags/ > cd
Current path is: /alice/cern.ch/user/p/psaiz/
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > mkdir taggedDir
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/ > cd taggedDir/
Current path is: /alice/cern.ch/user/p/psaiz/taggedDir/
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > addTag

Error: not enough arguments in addTag
Usage: affTag &lt directory> &lt tag name>
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > addTag . MonteCarloRuns
Jan 17 11:15 info Getting the file soap://aliendb.cern.ch:8092/home/alienMaster/catalogue/alice/tags/Monte
CarloRuns?URI=SE
And the file is /tmp/AliEn/cache/MonteCarloRuns.41221042798066
Creating the table T123030VMonteCarloRuns...
Tag created
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ >

Now, for each entry in the directory, you can enter the metadata with the command addTagValue. In this example, we are going to copy a file in the

directory that we just created, and we are going to give it some metadata.

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > cp /alice/simulation/2002-05/V3.08.Rev.02/00020/000 \
40/galice.root galice.root

Copying /alice/simulation/2002-05/V3.08.Rev.02/00020/00040/galice.root to /alice/cern.ch/user/p/psaiz/taggedDir/gal \
ice.root...
file is /alice/cern.ch/user/p/psaiz/taggedDir/galice.root.
File /alice/cern.ch/user/p/psaiz/taggedDir/galice.root inserted in the catalog
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > addTagValue
Error: not enough arguments in addValueTag
Usage: addTagValue &lt file> &lt tag> &lt variable>=&lt value> [&lt variable>=&ltvalue> ...]
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > addTagValue galice.root MonteCarloRuns GluonRadiati \
on=1 phimin=3.5

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > showTagValue . MonteCarloRuns
file(char(20)) offset(int(11)) eventGen(char(15)) GluonRadiation(int(1)) HolesPHOSRICH(int(1)) ...
[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ >

Searching for files

Like in UNIX file system, you can search for files in the catalogue. You have to give the directory where you want to start the search, and part of the name

you are looking for:

[aliendb.cern.ch:3307] /alice/cern.ch/user/p/psaiz/taggedDir/ > cd /alice/simulation/2001-02/V3.06/
Current path is: /alice/simulation/2001-02/V3.06/
[aliendb.cern.ch:3307] /alice/simulation/2001-02/V3.06/ > find . galice.root

Searching for files like /alice/simulation/2001-02/V3.06/*galice.root* ...
/alice/simulation/2001-02/V3.06/00001/00001/galice.root
/alice/simulation/2001-02/V3.06/00001/00002/galice.root
...
/alice/simulation/2001-02/V3.06/00050/00098/galice.root
/alice/simulation/2001-02/V3.06/00050/00100/galice.root
5676 files and directories found
[aliendb.cern.ch:3307] /alice/simulation/2001-02/V3.06/ >

But you can also make more complex queries, using the metadata. In that case, you also have to specify which tag you are using, and the condition

in the tag:

[aliendb.cern.ch:3307] /alice/simulation/2001-02/V3.06/ > find
Error: not enough arguments in find
Usage: find &lt path> &lt fileName> [[ &lt tagname>: &lt condition>] [ [and|or] [ &lt tagname>: &lt condition>]]*]
[aliendb.cern.ch:3307] /alice/simulation/2001-02/V3.06/ > find . galice MonteCarloRuns:GluonRadiation=0

Searching for files like /alice/simulation/2001-02/V3.06/*galice* ...
Filtering with tag MonteCarloRuns
/alice/simulation/2001-02/V3.06/00041/00001/galice.root
/alice/simulation/2001-02/V3.06/00041/00002/galice.root
/alice/simulation/2001-02/V3.06/00041/00003/galice.root
/alice/simulation/2001-02/V3.06/00041/00004/galice.root
...
/alice/simulation/2001-02/V3.06/00050/00098/galice.root
/alice/simulation/2001-02/V3.06/00050/00100/galice.root
973 files and directories found
[aliendb.cern.ch:3307] /alice/simulation/2001-02/V3.06/ >