Storage Element Specification

Quick Reference to use dynamic SE discovery with failover:
 

Specifying in a JDL...

to store an job's output file according to the VOs default policy (for ALICE 'disk=2'):

OutputArchive={”myArchive:file*”};
 
OutputFile={”file*”};

to save number of copies depending on QoS flags, using dynamic discovery and failover:

OutputArchive={”myArchive:fileA*@disk=4” , ”myArchive:fileB*@disk=2,tape=1"};
 
OutputFile={”fileA*@disk=4” , ”fileB*@disk=2,tape=1”};

 

Using the add command...

Default policy (for ALICE 'disk=2')
add fileLFN filePFN

Specify count of QoS flags, using dynamic discovery and failover:
add fileLFN filePFN    disk=2,tape=1

 

 

 

 

Detailed Feature Reference of the SE specification discovery:

 

Specifying the Output in a JDL:

OutputArchive    = {”  archive: file1, file2    @ <STORAGETAGS>    , <JobOutPutOptions>  ”};
OutputFile    =   {”  file3, file4, file5    @ <STORAGETAGS>    , <JobOutPutOptions>  ”};

<JobOutPutOptions> may be a comma separated combination of:

"no_links_registration"        The files in the archive will not be registered in the catalogue
(you will only see the archive, and in case have to extract files manually).

"no_archive"            files normally go to an archive, even if they are specified as files. with
this parameter they will actually be stored as files

 

 

Specifying to add or upload a file (a JobAgent uses upload to handle its output files).

upload      file-PFN # for UploadOptions see 'help upload'

add       file-LFN  file-PFN # for AddOptions see 'help add'

The Storage Element Specification by :

may be a comma separated combination of:

#    get the best N SEs for a QoS type (dynamic SE discovery)
#    a list of Storage Elements (use/not)
<SELIST>, #    a list of SEs to select N out of it

:     The SE Rank Cache will be asked for the topmost N
entries for a given QoS.

e.g.        disk=3
or           tape=1
or           disk=1,tape=1

If any SE fails, the Rank Cache will be requested to
deliver an adapted failover list of SEs and we won't give up
until all known+working SEs are tried out.

:    A list of SEs, that can be positive (implicit)
and negative. Negative exists in order to exclude SEs
from auto discovery (see Examples below).

e.g.    ALICE::NIHAM::FILE,  ALICE::SARA::DCACHE,  ALICE::SaoPaulo::DPM
or       ALICE::NIHAM::FILE,  !ALICE::SARA::DCACHE, ALICE::SaoPaulo::DPM
or      !ALICE::NIHAM::FILE, !ALICE::SARA::DCACHE, !ALICE::SaoPaulo::DPM

This positive SELIST specification will be considered as static.
There will be no auto discovery or replacement based on availability
or ranking and no retry or failover in case of error. Beyond, ensure you have the permissions to write to a Storage Element before specifying it explicitly, as access might be restricted to certain users (See Exclusive SEs).

→ If you specify a SE that will not work, the job will finish with
either status <SAVED_WARNING> or even <ERROR_SV>. In the first case, the JobAgent could store in at least one location, in the later it couldn't store in any location and the job output is lost.

:   Use any N elements of the list (optional combination with )

e.g.     ALICE::Catania::SE,ALICE::NIHAM::DCACHE,ALICE::SaoPaulo::DPM, select=2

If N < # of positive entries in SELIST, then it will try all possibilties
in case of error. Apart from that failback within the user specification,
it is static.

Remarks:

The definition of QoS flags is independent of the code. The actual flags are stored
in the LDAP entry of an SE. The ALICE VO currently uses 'disk' and 'tape' as flags
accessable for users. (see upcoming documentation about exclusive SE users).

In case of no user-side SE specification, an LDAP entry defines the default policy
of upload and add, which is currently 'disk=2' for the ALICE VO.

By whatever method, you only may specify up to 9 replicas/copies in total. A user
speficication is evaluated and just truncated, once it exceeds this limit.

Some Examples for JDLs:

Store an output file according to the VOs default policy (for ALICE 'disk=2'):
OutputFile={”file*”};

Specify count of QoS flags, using dynamic discovery and failover:
OutputFile={”fileA*@disk=4” , ”fileB*@disk=2,tape=1”};

 

Specify static SEs (Without availability or performance ranking and no failover!):
OutputFile={”file*@ALICE::EARTH::SE,ALICE::VENUS::SE”};

Speficy static SEs with select N out of list.
OutputFile={”file*@ALICE::MARS::SE,ALICE::JUPITER::SE,ALICE::PLUTO::SE,select=2”};

Specify to exclude certain SEs from dynamic discovery:
OutputFile={”file*@!ALICE::MARS::SE,!ALICE::PLUTO::SE,disk=2”};

Speficy static SEs, plus a count of QoS flags with dynamic discovery:
OutputFile={”file*@ALICE::PLUTO::SE,ALICE::JUPITER::SE,!ALICE::MARS::SE,disk=1”};

Full combination of possible features:
OutputFile={”file*@ALICE::MARS::SE,ALICE::VENUS::SE,select=1,!ALICE::JUPITER::SE,disk=2,tape=3”};

Finally, the entries are not required to be in any order:
OutputFile={”file*@tape=3,!ALICE::JUPITER::SE,select=1,ALICE::MARS::SE,ALICE::VENUS::SE,disk=2”};

For add and upload at the alien prompt, the syntax and semantics are the same.

Some examples for the command add:

Default policy (for ALICE 'disk=2')
add fileLFN filePFN

Specify count of QoS flags, using dynamic discovery and failover:
add fileLFN filePFN    disk=2,tape=1

Full combination of possible features:
add fileLFN filePFN    ALICE::PLUTO::SE,ALICE::JUPITER::SE,!ALICE::MARS::SE,disk=1