# Certificate Security Protocol (CSP)

In order to make the Grid Engine internal communication more secure, Grid Engine has been enhanced by a certificate based security layer that sits on top between the GDI and comlib layer. The security layer is based on OpenSSLs libcrypto and uses RSA for secret key exchange and rc4 (keylength 128bit) for encryption. The implementation originated from a diploma thesis realised in 1996. Instead of transfering messages in clear text, the messages are encrypted with a secret key. The secret key is exchanged via a public/private key protocol. The user presents its certificate to SGE to prove its identity and receives the certificate from SGE to be sure he is communicating to the correct system. After this initial announcement phase the communication is transparently continued in encrypted form. The session is valid only for a certain period then session has to be reannounced.
Beginning with GE 60u4 the handshake mechanism for exchanging a secret key is based on the standard SSL handshake mechanism.

Below you can find the steps to setup a CSP secured system.

### Installation and setup of CSP secured Grid Engine

The CSP mode is enabled in the product_mode file ($SGE_ROOT/{default | $SGE_CELL/common/product_mode) by adding the suffix -csp to sge or sgeee.

To setup the Certificate Security Protocol enhanced version of Grid Engine, the steps involved are very similar to the standard setup. Apart from the standard setup of Grid Engine the following additional steps are necessary:

- Generation of the CA system keys and certificates on the master machine
  This is done by calling the install procedures with the -csp flag
- Distribution of the system keys and certificates to the execution and submit hosts
  It is the task of the system administrator to do it in a secure way (the keys must be transmitted to the execution/submit hosts in a secure manner e.g via ssh.
- Generation of User keys and certificates
  This can be done automatically by the sysadmin after master installation.
- Admittance of new users by the sysadmin

The security enhancement can be either added to a previously configured system or the system can be setup to support CSP security during the installation. The installation is performed as usual except that the install script is called with the additional parameter -csp.

To generate the CSP certificates and keys the following information must be supplied:

Country code State Location Organization Organizational unit CA email address | C=US ST=California L=San Francisco O=8Bits OU=Support emailAddress=admin@eightbits.com

The screen shots of an example installation outline the steps that are performed. If the system is already installed, the procedure looks very similar. Instead of using install_qmaster and install_execd, the script $SGE_ROOT/util/sgeCA/sge_ca -init is used. The screens below appear in the same manner.

First the Certificate Authority is created. The approach taken here, is to have a Grid Engine specific CA at the master host. The directory structure that contains security relevant information consists of two parts. Under $SGE_ROOT/{default | $SGE_CELL}/common/sgeCA the publicly accessible CA and daemon certificate are stored. The corresponding private keys are stored under /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/private. User keys and certificates go into /var/sgeCA/{sge_service | port$COMM_PORT}/{default | $SGE_CELL}/userkeys/$USER.

    Initializing Certificate Authority (CA) for OpenSSL security framework
    ----------------------------------------------------------------------

    Creating /scratch2/eddy/sge_sec/default/common/sgeCA
    Creating /var/sgeCA/port6789/default
    Creating /scratch2/eddy/sge_sec/default/common/sgeCA/certs
    Creating /scratch2/eddy/sge_sec/default/common/sgeCA/crl
    Creating /scratch2/eddy/sge_sec/default/common/sgeCA/newcerts
    Creating /scratch2/eddy/sge_sec/default/common/sgeCA/serial
    Creating /scratch2/eddy/sge_sec/default/common/sgeCA/index.txt
    Creating /var/sgeCA/port6789/default/userkeys
    Creating /var/sgeCA/port6789/default/private
    Hit  to continue >>

After setting up the directory structure the CA specific certificate and private key are generated. We use either pseudo random data from a special file or if available /dev/random for seeding the PRNG (pseudo random number generator, see http://www.openssl.org/support/faq.html and http://www.cosy.sbg.ac.at/~andi for additional info on random numbers)

    Creating CA certificate and private key
    ---------------------------------------

    Please give some basic parameters to create the distinguished name (DN)
    for the certificates.

    We will ask for

       - the two letter country code
       - the state
       - the location, e.g city or your buildingcode
       - the organization (e.g. your company name)
       - the organizational unit, e.g. your department
       - the email address of the CA administrator (you!)

    Hit  to continue >>

    Please enter your two letter country code, e.g. >US< >> DE
    Please enter your state >> Bavaria
    Please enter your location, e.g city or buildingcode >> Regensburg
    Please enter the name of your organization >> Gridware
    Please enter your organizational unit, e.g. your department >> Griders
    Please enter the email address of the CA administrator >> admin@griders.org

    You selected the following basic data for the distinguished name of
    your certificates:

    Country code:         C=DE
    State:                ST=Bavaria
    Location:             L=Regensburg
    Organization:         O=Gridware
    Organizational unit:  OU=Griders
    CA email address:     emailAddress=admin@griders.org

    Do you want to use these data (y/n) [y] >>

    Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed<

    1513428 semi-random bytes loaded
    Creating CA certificate and private key

    Using configuration from /tmp/sge_ca14364.tmp
    Generating a 1024 bit RSA private key
    .....++++++
    ................++++++
    writing new private key to '/var/sgeCA/port6789/default/private/cakey.pem'
    -----
    Hit  to continue >>

After the installation of the CA infrastructure application and user certificates and private keys are created and signed by the CA for the admin user, for the pseudo daemon user and for the user root. Currently we use the outdated uniqueIdentifier field for tieing the Unix user name to the certificate, this will change in a future implementation.

    Creating Daemon certificate and key
    -----------------------------------

    Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/private/rand.seed<

    1513428 semi-random bytes loaded
    Using configuration from /tmp/sge_ca14364.tmp
    Generating a 1024 bit RSA private key
    ...............++++++
    ................++++++
    writing new private key to '/var/sgeCA/port6789/default/private/key.pem'
    -----
    Using configuration from /tmp/sge_ca14364.tmp
    Check that the request matches the signature
    Signature ok
    The Subjects Distinguished Name is as follows
    countryName           :PRINTABLE:'DE'
    stateOrProvinceName   :PRINTABLE:'Bavaria'
    localityName          :PRINTABLE:'Regensburg'
    organizationName      :PRINTABLE:'Gridware'
    organizationalUnitName:PRINTABLE:'Griders'
    uniqueIdentifier      :PRINTABLE:'root'
    commonName            :PRINTABLE:'SGE Daemon'
    emailAddress          :IA5STRING:'none'
    Certificate is to be certified until Mar  5 13:50:57 2003 GMT (365 days)

    Write out database with 1 new entries
    Data Base Updated
    created and signed certificate for SGE daemons
    Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkey
    s/root/rand.seed<

    1513428 semi-random bytes loaded
    Using configuration from /tmp/sge_ca14364.tmp
    Generating a 1024 bit RSA private key
    ............++++++
    .................++++++
    writing new private key to '/var/sgeCA/port6789/default/userkeys/root/key.pem'
    -----
    Using configuration from /tmp/sge_ca14364.tmp
    Check that the request matches the signature
    Signature ok
    The Subjects Distinguished Name is as follows
    countryName           :PRINTABLE:'DE'
    stateOrProvinceName   :PRINTABLE:'Bavaria'
    localityName          :PRINTABLE:'Regensburg'
    organizationName      :PRINTABLE:'Gridware'
    organizationalUnitName:PRINTABLE:'Griders'
    uniqueIdentifier      :PRINTABLE:'root'
    commonName            :PRINTABLE:'SGE install user'
    emailAddress          :IA5STRING:'none'
    Certificate is to be certified until Mar  5 13:50:59 2003 GMT (365 days)

    Write out database with 1 new entries
    Data Base Updated
    created and signed certificate for user >root< in >/var/sgeCA/port6789/default/userkeys/root<
    Creating RANDFILE from >/kernel/genunix< in >/var/sgeCA/port6789/default/userkeys/eddy/rand.seed<

    1513428 semi-random bytes loaded
    Using configuration from /tmp/sge_ca14364.tmp
    Generating a 1024 bit RSA private key
    .............++++++
    .....................................................++++++
    writing new private key to '/var/sgeCA/port6789/default/userkeys/eddy/key.pem'
    -----
    Using configuration from /tmp/sge_ca14364.tmp
    Check that the request matches the signature
    Signature ok
    The Subjects Distinguished Name is as follows
    countryName           :PRINTABLE:'DE'
    stateOrProvinceName   :PRINTABLE:'Bavaria'
    localityName          :PRINTABLE:'Regensburg'
    organizationName      :PRINTABLE:'Gridware'
    organizationalUnitName:PRINTABLE:'Griders'
    uniqueIdentifier      :PRINTABLE:'eddy'
    commonName            :PRINTABLE:'SGE admin user'
    emailAddress          :IA5STRING:'none'
    Certificate is to be certified until Mar  5 13:51:02 2003 GMT (365 days)

    Write out database with 1 new entries
    Data Base Updated
    created and signed certificate for user >eddy< in >/var/sgeCA/port6789/default/userkeys/eddy<
    Hit  to continue >>

The security related setup of sge_qmaster is now complete and the installation procedure continues as usual:

    SGEEE startup script
    --------------------

    Your system wide SGEEE startup script is installed as:

       "/scratch2/eddy/sge_sec/default/common/rcsge"

    Hit  to continue >>

If the shared filesystem is not secure enough to hold the CSP security information in a place that can be accessed by the execution daemons as well, it is necessary to transfer the directory containing the daemon's private key and the random file to the execution host. Here it is installed locally before the setup of the execution daemon can be performed.

    Copy the private keys to the execd host
    On the master machine as user root:
    # umask 077
    # (cd /; tar cvpf /var/sgeCA/port6789.tar /var/sgeCA/port6789/default)
    On the execution host machine as user root:
    # scp /var/sgeCA/port6789.tar@<master host> /
    # (cd /; tar xvpf /var/sgeCA/port6789.tar )
    Verify the file permissions:
    # ls -lR /var/sgeCA/port6789/
    /var/sgeCA/port6789/:
    total 2
    drwxr-xr-x   4 eddy     other        512 Mar  6 10:52 default
    /var/sgeCA/port6789/default:
    total 4
    drwx------   2 eddy     staff        512 Mar  6 10:53 private
    drwxr-xr-x   4 eddy     staff        512 Mar  6 10:54 userkeys
    /var/sgeCA/port6789/default/private:
    total 8
    -rw-------   1 eddy     staff        887 Mar  6 10:53 cakey.pem
    -rw-------   1 eddy     staff        887 Mar  6 10:53 key.pem
    -rw-------   1 eddy     staff       1024 Mar  6 10:54 rand.seed
    -rw-------   1 eddy     staff        761 Mar  6 10:53 req.pem
    /var/sgeCA/port6789/default/userkeys:
    total 4
    dr-x------   2 eddy     staff        512 Mar  6 10:54 eddy
    dr-x------   2 root     staff        512 Mar  6 10:54 root
    /var/sgeCA/port6789/default/userkeys/eddy:
    total 16
    -r--------   1 eddy     staff       3811 Mar  6 10:54 cert.pem
    -r--------   1 eddy     staff        887 Mar  6 10:54 key.pem
    -r--------   1 eddy     staff       2048 Mar  6 10:54 rand.seed
    -r--------   1 eddy     staff        769 Mar  6 10:54 req.pem
    /var/sgeCA/port6789/default/userkeys/root:
    total 16
    -r--------   1 root     staff       3805 Mar  6 10:54 cert.pem
    -r--------   1 root     staff        887 Mar  6 10:54 key.pem
    -r--------   1 root     staff       2048 Mar  6 10:53 rand.seed
    -r--------   1 root     staff        769 Mar  6 10:54 req.pem

    Continue with the installation of sge_execd
    # cd $SGE_ROOT
    # ./install_execd -csp

### Generation of user keys and certificates

In order to let users use the CSP secured system, the user must have access to a user specific certificate and private key. They are generated by the administrator logged in as the user root on the master machine. To generate certificates and private keys for the user it is most convenient to create a file of the following format and to execute the commands in the next box.

    Create a file containing the following information, e.g. myusers.txt

    eddy:Eddy Smith:eddy@griders.org
    sarah:Sarah Miller:sarah@griders.org
    leo:Leo Lion:leo@griders.org

    where the fields are:

    Unix user:Gecos field:email address

    As user root on the master machine do:

    % $SGE_ROOT/util/sgeCA/sge_ca -usercert myusers.txt

    % ls -l /var/sgeCA/port6789/default/userkeys
    dr-x------   2 eddy  staff        512 Mar  5 16:13 eddy
    dr-x------   2 sarah staff        512 Mar  5 16:13 sarah
    dr-x------   2 leo   staff        512 Mar  5 16:13 leo

    Every user can then install its security related files in $HOME/.sge by:

    % source $SGE_ROOT/default/common/settings.csh
    % $SGE_ROOT/util/sgeCA/sge_ca -copy
    Certificate and private key for user eddy have been installed

    For every Grid Engine installation a subdirectory for the corresponding
    COMMD_PORT number is installed:

    % ls -lR $HOME/.sge
    /home/eddy/.sge:
    total 2
    drwxr-xr-x   3 eddy staff        512 Mar  5 16:20 port6789

    /home/eddy/.sge/port6789:
    total 2
    drwxr-xr-x   4 eddy staff        512 Mar  5 16:20 default

    /home/eddy/.sge/port6789/default:
    total 4
    drwxr-xr-x   2 eddy staff        512 Mar  5 16:20 certs
    drwx------   2 eddy staff        512 Mar  5 16:20 private

    /home/eddy/.sge/port6789/default/certs:
    total 8
    -r--r--r--   1 eddy staff       3859 Mar  5 16:20 cert.pem

    /home/eddy/.sge/port6789/default/private:
    total 6
    -r--------   1 eddy staff        887 Mar  5 16:20 key.pem
    -r--------   1 eddy staff       2048 Mar  5 16:20 rand.seed

For checking and looking at certificates the following commands might be helpful.

    % setenv ARCH `$SGE_ROOT/util/arch`

    Display a certificate:

    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -in ~/.sge/port6789/default/certs/cert.pem -text

    Check issuer

    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -issuer -in ~/.sge/port6789/default/certs/cert.pem -noout
    issuer= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=SGE Certificate Authority/uniqueIdentifier=CA/uniqueIdentifier=eddy/Email=eddy@griders.org

    Check subject
    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -subject -in ~/.sge/port6789/default/certs/cert.pem -noout
    subject= /C=DE/ST=Bavaria/L=Regensburg/O=Griders Org/OU=Testsystem at port 6789/CN=eddy donetti/Email=eddy@gridders.org

    Show email of cert
    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -email -in ~/.sge/port6789/default/certs/cert.pem -noout
    eddy@griders.org

    Show validity
    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -dates -in ~/.sge/port6789/default/certs/cert.pem -noout
    notBefore=Sep 25 14:48:38 2001 GMT
    notAfter=Sep 25 14:48:38 2002 GMT

    Show fingerprint
    % $SGE_ROOT/utilbin/$ARCH/openssl x509 -fingerprint -in ~/.sge/port6789/default/certs/cert.pem -noout
    MD5 Fingerprint=F9:FA:AB:86:F3:71:E4:7F:18:82:78:7D:5E:51:7B:B5

For renewing certificates proceed as follows (this script has been introduced in GE 6.0u4 and GE 5.3p7):

    Change to $SGE_ROOT and become root on the master host (we assume $SGE_CELL being 'default'):
    # tcsh
    # source $SGE_ROOT/default/settings.csh

    edit $SGE_ROOT/util/sgeCA/renew_all_certs.csh, change the number of days the certificates are valid:
    ---
      # extend the validity of the CA certificate by
      set CADAYS = 365
      # extend the validity of the daemon certificate by
      set DAEMONDAYS = 365
      # extend the validity of the user certificate by
      set USERDAYS = 365
    ---

    run the changed script (default for all extension times are 365 days from the day the script is run)

    # util/sgeCA/renew_all_certs.csh

    Then you have to replace the old certificates against the new ones on all hosts that installed them locally (i.e. under /var/sgeCA/..., see above under
    execution daemon installation).
    If users have copied certificates and keys to $HOME/.sge they have to repeat $SGE_ROOT/util/sgeCA/sge_ca -copy to have access to the renewed certificates.

Copyright 2002 Sun Microsystems, Inc. All rights reserved.