EOS installation guide and notes

EOS installation guide and more information

This page aims to help the admins to proceed with the EOS installation in storage servers, and add some information (commands, etc) in order to configure or tune the installed instance.

 

Installation

Traditionally, the installation is done in SL[5-6] and CentOS[7] based machines, using the script 'eos-deploy', download here: eos-deploy

You can also follow official EOS documentation but you might miss some ALICE particularities: EOS docs

 

0) Cleanup

eos-deploy will remove eos packages, and install eos-cleaup, then run it, which will return 'most' of the things to the proper initial status needed for installation.

 

1) make sure there is no XRootD V4.X installed if you go for aquamarine release - if yes, uninstall!
rpm -e `rpm -qa | grep xrootd | awk '{printf("%s ", $1); }'`

 

2) mask xrootd* and libmicrohttpd* from EPEL and base and update (e.g. CentOS-*) yum repositories:

This should be alerted by eos-deploy automatically if the repos are not ok. Do:

in /etc/yum.repos.d/epel.repo
in /etc/yum.repos.d/epel-testing.repo

add to each section:

exclude=xrootd*,libmicrohttpd*

 

This must be done in all eos related servers (both MGMs and FSTs)
 
 
3) Run eos-deploy (by root)
 
The latest versions of the script will create a 'eos-deploy.timestamp.config' with the values you have introduced during the wizard.
 
Some notes:
 
# You should have host certificates in the managers, under /etc/grid-security with:
Permissions 600 on hostcert.pem
Permissions 400 on hostkey.pem

 

# For ALICE case, the instance name should contain alice, e.g.: 'alicesitename'

 

# Make sure you have ssh access to FSTs (and if using a second MGM, too)
# When asked how many filesystems and for a regexp, use the number of disks in your FSTs. See example:
 
volume-n02-v01/cern-01 32T  2.5T   30T   8% /data1
volume-n02-v02/cern-02 32T  2.4T   30T   8% /data2
volume-n02-v03/cern-03 32T  2.6T   30T   8% /data3
 
Here, you would provide eos-deploy, with '3' for number of filesystems, and the regexp would be 'data'.
 
# Ports

1094: XRootD MGM port (only on MGMs)
1095: XRootD FST port (only on FSTs)
1096: XRootD SYNC port (only on MGMs)
1097: XRootD MQ port (only on MGMs)

In principle not used by ALICE:

443: https X509 port (only on HTTPS gateways or MGM)
8443: https KRB5 port (only on HTTPS gateways or MGM)
8000: http port (only on MGMs)
8001: http port (only on FSTs)

The script open the ports needed for you, but is a temporary change. Please make those network changes permanent:

After running the script, you should have all necessary eos services running.

  • Some helpful EOS commands

# You can login to eos with:

[root@mmmartinmgm1 ~]# eos -b
EOS Console [root://localhost] |/>

You get back that console, where you can run eos commands, and in a shell.

 

# You can see your filesystems/disks:

 

EOS Console [root://localhost] |/> fs ls

#..........................................................................................................................................
#                   host (#...) #   id #           path #     schedgroup #         geotag #       boot # configstatus #      drain # active
#..........................................................................................................................................
mmmartinfst1.cern.ch (1095)      1    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      2    /var/eos/fs/1        default.1                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      3    /var/eos/fs/2        default.2                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      4    /var/eos/fs/3        default.3                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      5    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      6    /var/eos/fs/1        default.1                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      7    /var/eos/fs/2        default.2                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      8    /var/eos/fs/3        default.3                        booted             rw      nodrain   online

 

# Your spaces (default is the one created on the script)

EOS Console [root://localhost] |/> space ls
#------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#     type #           name #  groupsize #   groupmod #N(fs) #N(fs-rw) #sum(usedbytes) #sum(capacity) #capacity(rw) #nom.capacity #quota #balancing # threshold # converter #  ntx # active #intergroup
#------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
spaceview           default            0           24      8         8         18.08 G        67.36 G       26.56 G             0    off        off          20         off      2        0         off

 

# The same for the groups:

 

EOS Console [root://localhost] |/> group ls
#---------------------------------------------------------------------------------------------------------------------
#     type #           name #     status #nofs #dev(filled) #avg(filled) #sig(filled) #balancing #  bal-shd #drain-shd
#---------------------------------------------------------------------------------------------------------------------
groupview  default.0                  on     2         0.74        26.84         0.74 idle                0          0
groupview  default.1                  on     2         0.74        26.84         0.74 idle                0          0
groupview  default.2                  on     2         0.74        26.84         0.74 idle                0          0
groupview  default.3                  on     2         0.74        26.84         0.74 idle                0          0

 

# Common case: you want to use RAIN6 (the inner EOS Raid6 equivalent) and you don't have 6 servers, when you need at least 6 stripes.

You can group fs in the same group, for example, doing a group twice the size. You will to reorganize the layout of the storage.

 

Example: 3 servers with 4 fs:

 

EOS Console [root://localhost] |/> fs ls
#..........................................................................................................................................
#                   host (#...) #   id #           path #     schedgroup #         geotag #       boot # configstatus #      drain # active
#..........................................................................................................................................
mmmartinfst1.cern.ch (1095)      1    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      2    /var/eos/fs/1        default.1                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      3    /var/eos/fs/2        default.2                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      4    /var/eos/fs/3        default.3                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      5    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      6    /var/eos/fs/1        default.1                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      7    /var/eos/fs/2        default.2                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      8    /var/eos/fs/3        default.3                        booted             rw      nodrain   online

mmmartinfst3.cern.ch (1095)      9    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      10  /var/eos/fs/1        default.1                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      11  /var/eos/fs/2        default.2                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      12  /var/eos/fs/3        default.3                        booted             rw      nodrain   online

 

we want:

 

EOS Console [root://localhost] |/> fs ls
#..........................................................................................................................................
#                   host (#...) #   id #           path #     schedgroup #         geotag #       boot # configstatus #      drain # active
#..........................................................................................................................................
mmmartinfst1.cern.ch (1095)      1    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      2    /var/eos/fs/1        default.0                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      3    /var/eos/fs/2        default.1                        booted             rw      nodrain   online
mmmartinfst1.cern.ch (1095)      4    /var/eos/fs/3        default.1                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      5    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      6    /var/eos/fs/1        default.0                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      7    /var/eos/fs/2        default.1                        booted             rw      nodrain   online
mmmartinfst2.cern.ch (1095)      8    /var/eos/fs/3        default.1                        booted             rw      nodrain   online

mmmartinfst3.cern.ch (1095)      9    /var/eos/fs/0        default.0                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      10  /var/eos/fs/1        default.0                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      11  /var/eos/fs/2        default.1                        booted             rw      nodrain   online
mmmartinfst3.cern.ch (1095)      12  /var/eos/fs/3        default.1                        booted             rw      nodrain   online

 

for that, we use these commands:

 

EOS Console [root://localhost] |/eos/alicemiguel/grid/> fs mv 'fs_id' default.'group_id'
success: moved filesystem 'id' into space default.'gid'

 

Then we set the number of stripes:

 

|eos> attr set sys.forced.stripes=6 /eos/instancename/foldername

 

That last command also offers other possibilities. You can always use the help of each command.

EOS Console [root://localhost] |/> attr ls

to see all possibilities.

 

# File information

EOS Console [root://localhost] |/eos/alicemiguel/grid/> file info passwd
File: '/eos/alicemiguel/grid/passwd'  Flags: 0640
Size: 1372
Modify: Wed Feb  4 17:32:57 2015 Timestamp: 1423067577.602053000
Change: Wed Feb  4 17:32:58 2015 Timestamp: 1423067578.87207739
CUid: 0 CGid: 0  Fxid: 00000006 Fid: 6    Pid: 11   Pxid: 0000000b
XStype: none    XS:     ETAG: 1610612736:1423067577
plain Stripes: 1 Blocksize: 4k LayoutId: 00100001
#Rep: 1
#   fs-id  #...................................................................................................................................
#                   host  #     schedgroup #           path #     boot # configstatus #      drain # active #                 geotag
#...................................................................................................................................
0       3    mmmartin-fst1.cern.ch         default.2    /var/eos/fs/2     booted             rw      nodrain   online                         
*******

 

# Copy a file

EOS Console [root://localhost] |/eos/alicemmmartin/test/> cp /etc/hosts /eos/alicemmmartin/test/
doing stat of /etc/hosts
[eos-cp] going to copy 1 files and 159 B
append: /etc/hosts hosts

[eoscp] hosts                    Total 0.00 MB  |====================| 100.00 % [0.0 MB/s]
[eos-cp] copied 1/1 files and 159 B in 0.08 seconds with 2101 B/s

 

# EOSHA

This service tests that the MGM is running every 10s. If it fails, it will send an email to the configured admin /etc/sysconfig/eos.

 

# Add new FST

In FST:
  eosfstregister mgm-host-name /regex/path default:nfs
e.g.:
  eosfstregister mmmartinmgm1.cern.ch /data default:4
 
being nfs the number of fs you have
 
You have to enable the new node on the MGM:
  eos node set fst-host-name:1095 on
 
e.g.:

  eos node set mmmartinfst1.cern.ch:1095 on

 

 

  • Some testing

 

06:18:10 # eos cp /var/tmp/1G /eos/aliceornl/test/
doing stat of /var/tmp/1G
[eos-cp] going to copy 1 files and 1.05 GB
append: /var/tmp/1G 1G

[eoscp] 1G                       Total 1000.00 MB    |====================| 100.00 % [403.5 MB/s]
[eos-cp] copied 1/1 files and 1.05 GB in 2.64 seconds with 396.77 MB/s

_________________________________________________________

09:47:20 # eoscp -b 100000000 root://localhost//eos/aliceornl/test/1G /dev/null
[eoscp]                          Total 1000.00 MB    |====================| 100.00 % [754.4 MB/s]
[eoscp] #################################################################
[eoscp] # Date                     : ( 1431881243 ) Sun May 17 09:47:23 2015[eoscp} # auth forced= krb5= gsi=
[eoscp] # Source Name [00]         : root://localhost//eos/aliceornl/test/1G
[eoscp] # Destination Name [00]    : /dev/null
[eoscp] # Data Copied [bytes]      : 1048576000
[eoscp] # Realtime [s]             : 1.390000
[eoscp] # Eff.Copy. Rate[MB/s]     : 754.371250
[eoscp] # Write Start Position     : 0
[eoscp] # Write Stop  Position     : 1048576000

_________________________________________________________

xrdcp  root://localhost//eos/aliceornl/test/1G /dev/null -f
[xrootd] Total 1000.00 MB    |====================| 100.00 % [699.1 MB/s]

 

If there is an issue with permissions, use ruid=0 parameter to map to 'nobody', like:

xrdcp  "root://localhost//eos/aliceornl/test/1G?ruid=0" /dev/null -f

 

_________________________________________________________

root@alice-eos-01.ornl.gov:/eos/aliceornl/test
10:29:49 # time for name in `seq 1 1000`; do touch empty.$name; done

real    0m11.058s
user    0m0.409s
sys    0m0.882s

_________________________________________________________

EOS Console [root://localhost] |/eos/aliceornl/test/> mkdir /test
EOS Console [root://localhost] |/eos/aliceornl/test/> test mkdir 1000
info: doing directory test with loop =1000
[       mkdir]        startstop         : 771.510
=       mkdir=        startstop         : 771.510
EOS Console [root://localhost] |/eos/aliceornl/test/> test rmdir 1000
info: doing directory test with loop =1000
[       rmdir]        startstop         : 711.595
=       rmdir=        startstop         : 711.595

 

 

 

! Important: BACKUP of MGM !

You should backup /var/eos/md/ daily. You should backup once the final config /var/eos/config/default.eoscf
Or you run a standby MGM and use the EOS sync service for a in-time replication of the /var/eos/md files