ResourceQuotaSpecification

# Specification Resource Quota

Roland Dittel
20 November 2006

## 1 Introduction

In large enterprise clusters it is necessary to prevent users from consuming all available resources. In order to achieve this, N1GE6 supports complex attributes which can be configured on a global, queue or host layer. This feature is sufficient in certain cases, especially in small clusters, but has shortcomings and drawbacks for enterprise usage.

Customers have asked for a feature to enhance resource limits so that they apply to several kinds of resources, several kinds of resource consumers, to all jobs in the cluster and to combinations of consumers. In this context, "resources" are any defined complex attribute (see complex(5)) known by the Grid Engine configuration. For example this can be slots, arch, mem_total, num_proc, swap_total or any custom-defined resource like compiler_license. Resource consumers are (per) users, (per) queues, (per) hosts, (per) projects, (per) parallel environments. This specification describes a user interface to define such flexible resource limits.

This feature provides a way for administrators to limit the resources used at a single time by a consumer. However, it is not a way to define priorities by which user should obtain a resource. Priorities can be defined by using the Share Tree feature released with N1GE6.

## 2 Project Overview

### 2.1 Project Aim

The aim of this project is a solution that allows utilization of built-in and user-defined resources to be managed in a more flexible manner. In particular, this is a means to limit resources on a per user basis and a per project basis. Similarly, resource limitations on the basis of a user groups and project groups are also required.

The Issues targeted with this project are:

Issue | Description
--- | ---
74 | Support maxujobs on a per host level
1532 | Max jobs per user on a queue basis
1644 | Per-user slot limits for limiting PE usage
CR 6298406 | Hostgroups should be added as another configuration layer b/w global and host
CR 6289250 | Request for Job limit per User of Queue

### 2.2 Project Benefit

The expectation is that the management of N1GE cluster resources will be possible in a much more targeted manner. The enhancement must make it easy to freely manage limits for arbitrary resources in relation to existing N1GE objects, such as project/user/host/queue/pe, without the burden of doing micro-management with countless projects/users/hosts/queues.

Suggestions for future enhancements are:

- express these resource limits by means of percentages of a wider context (e.g. (a) memory limit of 4G available for project1 and project2 (b) up to 70 percent available for project1 and (c) up to 60 percent available for project2)
- add a new built-in complex attribute "jobs", that always counts 1 for a job in all resource containers
- define operators which can modify a set of resource limits as a means to allow hierarchical management

## 4 Functional Definition

### 4.1 Performance

### 4.2 Reliability, Availability, Serviceability (RAS)

- Enhancement of Scheduler Profiling

### 4.3 Diagnostics

- qquota (new command)
- qstat -j job_id (enhancm.)

### 4.4 User Experience

#### 4.4.1 Obsolete Configuration

#### 4.4.2 Command Line (CLI)

##### 4.4.2.1 CLI enhancements

- qconf switches:

switch | Description
--- | ---
-aattr obj_nm attr_nm val obj_id_lst | add to a list attribute of an object
-Aaatr obj_nm fname obj_id_lst | add to a list attribute of an object
-dattr obj_nm attr_nm val obj_id_lst | delete from a list attribute of an object
-Dattr obj_nm fname obj_id_lst | delete from a list attribute of an object
-mattr obj_nm attr_nm val obj_id_lst | modify an attribute (or element in a sublist) of an object
-Mattr obj_nm fname obj_id_lst | modify an attribute (or element in a sublist) of an object
-rattr obj_nm attr_nm val obj_id_lst | replace an attribute (or element in a sublist) of an object
-Rattr obj_nm fname obj_id_lst | replace an attribute (or element in a sublist) of an object
obj_nm | rqs - resource quota set
attr_nm | name or enabled or description or limit
val | new value of attr_nm
obj_id_lst | rule set or rule for limit

- qstat switches:

switch | description
--- | ---
-j job_identifier_list | show scheduler job information
-u user_list | view only jobs of this user

##### 4.4.2.2 CLI additions

- qconf switches:

switch | Description
--- | ---
-arqs [name] | add resource quota set(s)
-Arqs fname | add resource quota set(s) from file
-mrqs [name] | modify resource quota set(s)
-Mrqs fname [name] | modify resource quota set(s) from file
-srqs [name_list] | show resource quota set(s)
-srqsl | show resource quota set list
-drqs [name_list] | delete resource quota set(s)

- qquota switches:

switch | description
--- | ---
-help | print this help
-h host_list | display only selected host
-l resource_attributes | request the given resources
-u user_list | display only selected users
-pe pe_list | display only selected parallel environments
-P project_list | display only selected projects
-q wc_queue_list | display only selected queues
-xml | display the information in XML-Format

#### 4.4.3 Graphical User Interface (GUI)

##### 4.4.3.1 Configuration

Qmon will be enhanced to allow the configuration of resource resource quota sets. The configuration will be the same as on CLI with an editor.

##### 4.4.3.2 Diagnose

No Diagnose Support will be provided by qmon

### 4.5 Manufacturing

- aimk - not affected
- Makefiles - minor changes, new source objects

### 4.6 Quality Assurance

- New testsuite tests needed
- Modules tests for core module testing

### 4.7 Security & Privacy

Not affected

### 4.8 Migration Path

- imposes no need for DB update
- imposes no need for update script

### 4.9 Documentation

This Specification is used by Doc writer.

#### 4.9.1 Man Page Changes:

- qquota(1) - new man page
- sge_resource_quota(5) - new man page
- qconf(5) - new switches

### 4.10 Installation

Installation will not not change. For future releases the installation may change if complex configuration for global/queue/host becomes obsolete.

At installation time no default rules sets are created.

### 4.11 Packaging

Does not change

## 5 Component Descriptions

### 5.1 Component Resource Quota Rules

#### 5.1.1 Overview

According to customers and the filed RFEs it's desired to define a limit only for specific consumers like users or projects and only for specific providers like hosts or queues. To achieve this administrators must be able to define a rule set which consists of the limiting resource and the limit value, and additionally the consumers or providers to whom this rule should apply. Because every rule can be expressed by a tuple of filter specifiers we decided to implement the rule sets in style of firewall rules.

In practice a rule is defined by:

- who
  - users (list of user or usersets/departments)
  - projects (list of project)
- where
  - parallel_environments (list of pe's)
  - hosts (list of host or hostgroups)
  - queues (list of cluster queues)
- what
  - resource_attribute=max value

The Resource Quota Rules are separate configuration objects and only used for scheduling decisions. They don't affect the overall cluster configuration like cluster queues, hosts or projects.

Deliberate use of restrictions in first step of implementation:

- Limits are counted per task as done in the current implementation. For example if a pe job got 10 slots it will consume 10 licenses.
- Limitation can only be done for fixed and consumable resources, not for load values.
  future enhancement: see "Migration Path". That means to have resource configuration for load values in either global, host for queue level.

#### 5.1.2 Functionality

##### Integration with current implementation

The Resource Quotas are an addition to the current global, host and queue instance based scheduling order. The old implementation is still valid and can be used without the new rules. The rules enhances the old implementation and adds a new order layer on top of global to define a more precise limitation.

The implications of the layer order on resources are described in complexes(5) under "Overriding attributes". In general the layers are AND associated and if one layer denies the job, then the next layer is ignored. For example, a limit value of "slots=4" can be overwritten in global, host or queue layer if the layer value is more restrictive, eg, "slots=2". The exception (see complexes(5)) is for boolean values; for example "is_linux=true" defined in the tree can not be overwritten to "is_linux=false" in global host or queue definition.

    resource quotas
      |-DENIED->break
      |
    global
      |-DENIED->break
      |
    host
      |-DENIED->break
      |
    queue
      |-DENIED->break
      |
     OK

##### Resource Reservation

Resource Reservation will for Resource Quotas analogue to the current global/host/queue resource configuration. No changes on client side necessary.

#### 5.1.3 Interfaces

##### Resource Quota Set Syntax

    ALL:          '*'
    SEPARATOR:    ','
    STRING:       [^\n]*
    QUOTE:        '\"'
    S_EXPANDER:   '{'
    E_EXPANDER:   '}'
    NOT:          '!'
    BOOL:         [tT][rR][uU][eE]
                  | 1
                  | [fF][aA][lL][sS][eE]
                  | 0
    NAME:         [a-zA-Z][a-zA-Z0-9_-]*
    LISTVALUE:    ALL | [NOT]STRING
    LIST:         LISTVALUE [SEPARATOR VALUE]*
    NOTSCOPE:     LIST | S_EXPANDER LIST E_EXPANDER
    SCOPE:        ALL | STRING [SEPARATOR STRING]*
    RESOURCEPAIR: STRING=STRING
    RESOURCE:     RESOURCEPAIR [SEPARATOR RESOURCEPAIR]*

    rule: "limit" ["name" NAME] ["users" NOTSCOPE] ["projects" SCOPE] ["pes" SCOPE] \
    ["queues" SCOPE] ["hosts" NOTSCOPE] "to" RESOURCE NL

    ruleset_attributes: ("name" NAME NL)
                        ("enabled" BOOL NL)?
                        ("description" QUOTE STRING QUOTE)?

    ruleset: "{"
             (ruleset_attributes)
             (rule)+
             "} NL"

    rulesets: (ruleset)*

##### Resource Quota Sets Format

###### users

Contains a comma separated list of UNIX users or ACLs (see access_list(5)). This parameter filters for jobs by a user in the list or one of the ACLs in the list. Any user not in the list will not be considered for the resource quota. The default value is '*' which means any user. An ACL is differentiated from a UNIX user name by prefixing the ACL name with an '@' sign. To exclude a user or ACL from the rule the name can be prefixed with the '!' sign. Defined UNIX user or ACL names need not be known in the Grid Engine Configuration.

###### projects

Contains a comma separated list of projects (see project(5)). This parameter filters for jobs requesting a project of the list. Any project not in the list will not be considered for the resource quota. If no project filter is specified all projects and jobs with no requested project matches the rule. The value '*' means all jobs with requested projects. To exclude a project from the rule the name can be prefixed with the '!' sign. The value '!*' means only jobs with no project requested.

###### pes

Contains a comma separated list of PEs (see sge_pe(5)). This parameter filters for jobs requesting a pe of the list. Any PE not in the list will not be considered for the resource quota. If no pe filter is specified all pe and jobs with no requested pe matches the rule. The value '*' means all jobs with requested pe. To exclude a pe from the rule the name can be prefixed with the '!' sign. The value '!*' means only jobs with no pe requested.

###### queues

Contains a comma separated list of cluster queues (see queue_conf(5)). This parameter filters for jobs may be scheduled in a queue of the list. Any queue not in the list will not considered be for the resource quota. The default value is '*' which means any queue. To exclude a queue from the rule the name can be prefixed with the '!' sign.

###### hosts

Contains a comma separated list of host or hostgroups (see host(5) and hostgroup(5)). This parameter filters for jobs may be scheduled on a host of the list or a host contained in the hostgroup. Any host not in the list will not be considered for the resource quota. The default value is '*' which means any hosts. To exclude a host or hostgroup from the rule, the name can be prefixed with the '!' sign.

##### Basic Configuration

###### Single Resource Quota Rule

Resource Quota rules specify the filter criteria that a job must match and the resulting limit that is taken when a match is found.

A rule must always begin with the keyword "limit". The order of the filter criteria is not important to define and input a rule. After sending the new rule set to the qmaster the rules will be ordered automatically to a human readable form.

###### Scope Lists

To define a rule for more than one filter scope, it is possible to group scopes to a list. The defined resource limit counts for all objects listed in the scope in sum.
For example we have a consumable virtual_free defined as:

    #name               shortcut   type        relop requestable consumable default  urgency
    #----------------------------------------------------------------------------------------
    virtual_free        vf         MEMORY      <=    YES         YES        1g       0

In the rule defined below, both users can use together only 5g of virtual_free:

      limit users roland, andre to virtual_free=5g

If the administrator wants to limit each of the two users to 5g virtual_free he could define two rules:

      limit users roland to virtual_free=5g
      limit users andre to virtual_free=5g

This is very cumbersome for large numbers of users or user groups. For this case a rule can be defined with an expanded list. This would look like:

      limit users {roland, andre} to virtual_free=5g

If the scope contains a usergroup then it gets also expanded and the limit counts also for each member of that group.
For example if a hostgroup @lx_hosts contains host durin and carc both rules are equivalent:

    1)
      limit users * hosts durin to virtual_free=10g
      limit users * hosts carc to virtual_free=10g
    2)
      limit users * hosts {@lx_hosts} to virtual_free=10g

###### NOT Operator

Sometimes it is necessary to define a rule for a userset but exclude some users of that set. This can be defined by using the NOT operator ('!' sign) in front of the user name. A rule so defined will not affect the excluded user, even if the user is explicitly added to the rule.

For example, user "roland" is also member of usergroup "staff". If a resource quota rule looks like this:

      limit users @staff,!roland to slots=10

the limit will not be effective for user "roland". Even if the resource quota rule looks like this:

      limit users @staff,!roland,roland to slots=10

the rule will not be effective for user "roland"

###### Dynamical Limits

Resource Quota rules always define a maximal value of a resource that can be used. In the most cases these values are static and equal for all matching filter scopes. If administrators want different rule limits on different scopes then they have to define multiple rules; this leads to a duplication of nearly identical rules. With the concept of dynamical limits this kind of duplication can be avoided.

A dynamical limit is a simple algebraic expression used to derive the rule limit value. To be dynamical the formula can reference a complex attribute whose value is used for the calculation of the resulting limit. The limit formula expression syntax is that of a summation weighted complex values, that is:

       {w1|$complex1[*w1]}[{+|-}{w2|$complex2[*w2]}[{+|-}...]]

Note, no blanks are allowed in the limit formula.

The following example clarifies the use of dynamical limits: Users are allowed to use 5 slots per CPU on all linux hosts.

      limit hosts {@linux_hosts} to slots=$num_proc*5

The complex attribute num_proc is defined on all hosts and its value is the processor count on every host. The limit is calculated by the formula "$num_proc*5" and so is different on some hosts. On a 2 CPU host users can run 10 slots whereas on a 1 CPU host users only can run 5 slots.

To be able to set the limitation to a well-defined value some prerequisites must be fulfilled

- limit formulas are only possible for INT and DOUBLE.
- The complex must be already defined in the complex list
- The complex must be defined either on global, or queue, or host layer to resolve the value
- The limitation complex must be the same value definition as the referenced complex value definition (for example slots=INT, num_proc=INT)
- The resource quota rule must be defined as an expanded list for the layer the complex is defined. (for example hosts {*} for $num_proc reference) It's not allowed to reference a complex for a sum of scopes (for example hosts * for $num_proc).

In principle all INT or DOUBLE kind of complex values could be referenced but due to time constrains the first implementation allows only $num_proc in combination with an expanded host list.

###### Resource Quota Rules and Resource Quota Set Interaction

In practice administrators define some global limits and some limits that only apply for some resource consumers. These resource quota rules are equitable. But in some cases it's necessary to define exceptions for some resource consumers. These resource quota rules are not equal and dominate some others. As a matter of that fact it is necessary to allow the definition of a prioritized rule list and a rule list that apply all of the time. This is done by grouping one or more singe rules into a number of rule sets.

Inside one rule set the rules are ordered and the first rule found is used. This is analogous to firewall rules and generally understood by administrators and allows the prioritization of some rules. A rule set always results in one or none effective resource quota for a specific request.

All of the configured rule sets apply all of the time. This means if multiple rule sets are defined the most restrictive set is used and allows to define equitable limits.

The following example clarifies the combination of rules and rule sets. We have a consumable defined as:

    #name               shortcut   type        relop requestable consumable default  urgency
    #----------------------------------------------------------------------------------------
    compiler_lic        cl         INT         <=    YES         YES        0        0

The resource quota sets are defined as:

    {
      name ruleset1
      limit users roland to compiler_lic=3
      limit projects * to compiler_lic=2
      limit users * to compiler_lic=1

    }
    {
      name ruleset2
      limit users * to compiler_lic=20
    }

The first rule set ruleset1 express:

- user roland is allowed to use 3 compiler_lic resources
- any request submitted in a project is allowed to use 2 compiler_lic resources
- the default value for all other users is 1 compiler_lic resource

The second rule set ruleset2 express:

- all requests together are only allowed to use 20 compiler_lic resources.

Inside ruleset1 the priority is clear defined, user roland will always get 3 compiler_lic resources even though he matches to "users *" of the last rule in the rule set and even if he would submit his request in a project. Also the interaction between ruleset1 and ruleset2 is clear defined and results in a reject if 20 compiler_lic resources are already in use, even if user roland does not use all of his 2 compiler_lic resources.

##### CLI - Command Line Interface

###### qconf

With qconf it is possible to edit the rule sets in an editor session like with the most qconf switches. To reduce the amount of data presented to the administrator its possible to select only one rule set for editing.

It's not possible to edit single rules. Because the rules inside the rule set are ordered, the meaning of a single rule depends on the context of all other rules. Therefore it doesn't make sense to edit a single rule without presenting the context of the rule.

Switch Descriptions:

- -Arqs fname (add RQS configuration)

Add the resource quota set (RQS) defined in fname to the Grid Engine cluster. Returns 0 on success and 1 if rqs is already defined. Requires root or manager privileges.

    $ more rule_set.txt
    {
      name rule_set_2
      enabled true
      description "rule set 2"
    }

    $ qconf -Arqs rule_set.txt
    rd141302@es-ergb01-01 added "rule_set_2" to resource quota set list

    $ qconf -Arqs rule_set.txt
    resource quota set "rule_set_2" already exists

- -Mrqs fname [rqs_name] (modify RQS configuration)

Same as -mrqs (see below) but instead of invoking an editor to modify the RQS configuration the file fname is considered to contain a changed configuration. The name of the rule set in fname must be the same as rqs_name. If rqs_name is empty all rule sets are overwritten by the rule sets in fname. Refer to sge_rqs(5) for details on the RQS configuration format. Returns 0 on success and 1 on error. Requires root or manager privilege.

    $ more rule_set.txt
    {
      name rule_set_3
      enabled true
      description "rule set 2"
    }

    $ qconf -Mrqs rule_set.txt rule_set_3
    resource quota set "rule_set_3" does not exist

    $ qconf -Mrqs rule_set.txt rule_set_4
    resource quota set "rule_set_4" does not match rule set definition

    $ qconf -Mrqs rule_set.txt
    rd141302@es-ergb01-01 modified resource quota set list

- -arqs [name] (add new RQS)

Adds a Resource Quota Set (RQS) description under the name lsr_name to the list of RQSs maintained by Grid Engine (see sge_rqs(5) for details on the format of a RQS definition). Qconf retrieves a default RQS configuration and executes vi(1) (or $EDITOR if the EDITOR environment variable is set) to allow you to customize the RQS configuration. Upon exit from the editor, the RQS is registered with sge_qmaster(8). Returns 0 on success and 1 if rqs is already defined. Requires root/manager privileges.

    $ qconf -arqs
    <- {
    <-   name template
    <-   enabled true
    <-   description ""
    <- }
    -> :q
    resource quota set name "template" is not valid

    $ qconf -arqs rule_set_1
    <- {
    <-   name rule_set_1
    <-   enabled true
    <-   description ""
    <- }
    -> :wq
    rd141302@es-ergb01-01 added "rule_set_1" to resource quota set list

    $ qconf -arqs rule_set_1
    resource quota set "rule_set_1" already exists

- -mrqs [name] (modify RQS configuration)

Retrieves the whole rules or only the specified current configuration for the resource quota set (RQS), executes an editor (either vi(1) or the editor indicated by the EDITOR environment variable) and registers the new configuration with the sge_qmaster(8). Refer to sge_rqs(5) for details on the RQS configuration format. Returns 0 on success and 1 on error. Requires root or manager privilege.

    $ qconf -mrqs rule_set_1
    <- {
    <-   name rule_set_1
    <-   enabled true
    <-   description ""
    <- }
    -> :wq
    rd141302@es-ergb01-01 modified "rule_set_1" in resource quota set list

    $ qconf -mrqs unknown_set
    resource quota set "unknown_set" does not exist

    $ qconf -mrqs
    <- ...
    <-   name rule_set_1
    <- ...
    <-   name rule_set_2
    <- ...

- -srqs [name_list] (show RQS configuration)

Show the definition of the resource quota set (RQS) specified by the argument.

    $ qconf -srqs
    ...
      name rule_set_1
    ...
      name rule_set_2
    ...

    $ qconf -srqs rule_set_1
    ...
      name rule_set_1
    ...

- -srqsl (show resource quota sets list)

Show a list of the names of all resource quota sets currently configured.

- -drqs name_list (delete RQS)

Deletes the specified resource quota sets (RQS). Returns 0 on success and 1 if rqs_name is unknown. Requires root/manager privileges.

    $ qconf -drqs rule_set_1
    rd141302@es-ergb01-01 removed "rule_set_1" from resource quota set list

    $ qconf -drqs unknown_rule_set
    denied: resource quota set "unknown_rule_set" does not exist

    $ qconf -drqs
    rd141302@es-ergb01-01 removed resource quota set list

- -aattr obj_nm attr_nm val obj_id_lst

See qconf(1)

    $ qconf -srqs ruleset_1
    {
      name ruleset_1
      enabled true
      limit users @eng to slots=10
      limit name arch_rule users @eng to arch=lx24-amd64
    }

    $ qconf -aattr resource_quota limit slots=20 ruleset_1/1
    No modification because "slots" already exists in "limit" of "ruleset_1/1"

    $ qconf -aattr resource_quota limit compiler_lic=5 rule_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in rqs list

    $ qconf -aattr resource_quota limit arch=sol-sparc64 rule_1/arch_rule
    No modification because "arch" already exists in "limit" of "ruleset_1/1"

- -Aaatr obj_nm fname obj_id_lst

See qconf(1)

    $ more resource.txt
    limit slots=20

    $ qconf -Aattr resource_quota resource.txt ruleset_1/1
    No modification because "slots" already exists in "limit" of "ruleset_1/1"

    $ more resource2.txt
    limit compiler_lic=5

    $ qconf -Aattr resource_quota resource2.txt ruleset_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in resource_quota list

- -dattr obj_nm attr_nm val obj_id_lst

See qconf(1)

    $ qconf -dattr resource_quota limit compiler_lic=5 rule_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in rqs list

    $ qconf -dattr resource_quota limit compiler_lic=5 rule_1/1
    "compiler_lic" does not exist in "limit" of "resource_quota"

- -Dattr obj_nm fname obj_id_lst

See qconf(1)

    $ more resource.txt
    limit compiler_lic=20

    $ qconf -Dattr resource_quota resource.txt rule_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in resource_quota list

    $ qconf -Dattr resource_quota resource.txt rule_1/1
    "compiler_lic" does not exist in "limit" of "resource_quota"

- -mattr obj_nm attr_nm val obj_id_lst

See qconf(1)

    $ qconf -mattr resource_quota limit slots=5 rule_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in resource_quota list

    $ qconf -mattr resource_quota limit new_resource=5 rule_1/1
    Unable to find "new_resource" in "limit" of "resource_quota" - Adding new element.

    $ qconf -mattr resource_quota enabled false rule_1
    rd141302@es-ergb01-01 modified "ruleset_1" in resource_quota list

- -Mattr obj_nm fname obj_id_lst

See qconf(1)

    $ more resource.txt
    limit slots=20

    $ qconf -Mattr resource_quota resource.txt ruleset_1/1
    rd141302@es-ergb01-01 modified "ruleset_1/1" in resource_quota list

    $ more resource2.txt
    limit new_resource=5

    $ qconf -Mattr resource_quota resource2.txt ruleset_1/1
    Unable to find "new_resource" in "limit" of "resource_quota" - Adding new element.

- -rattr obj_nm attr_nm val obj_id_lst

See qconf(1)

- -Rattr obj_nm fname obj_id_lst

See qconf(1)

###### qstat

Switch Descriptions:

- -j job_identifier_list

Additional Output

    Example:
       cannot run on cluster because exceeds limit in rule_set_1
       cannot run on host "bla" because exceeds limit in rule_set_1
       cannot run on queue instance "all.q@host" because exceeds limit in rule_set_1

- -u user_list

To be consistent with qquota the default value of user_list changes from * (all users) to the calling user.

###### qquota

The qquota command is a diagnose tool for the resource resource quotas. The output is a table with the following rows:

    resource quota rule |    limit  |               filter

For each matched rule per rule set a line is printed if the usage count is not 0 for this rule. If one rule contains more than one resource attribute then one line is printed per resource attribute. By default it shows the effective limits for the calling user and for all other filter criteria like project or pe the wildcard "*" is used which means not explicit is used.

The output for the limit table is:

- for consumable resources

       complex=used/limit (for example slots 2/20)

- for static resources

       complex=value (for example arch lx24-amd64)

The administrator and the user may define files (analogue to sge_qstat(5)), which can contain any of the options described below. A cluster-wide sge_qquota file may be placed under $SGE_ROOT/$SGE_CELL/common/sge_qquota The user private file is searched at the location $HOME/.sge_qquota. The home directory request file has the highest precedence over the cluster global file. Command line can be used to override the flags contained in the files.

Example: All users together should never take more than 20 slots All users should maximal take 5 slots on all linux hosts Every user is restricted to one slot per linux host, only user "roland" is restricted to 2 slots and all other slots on hosts are set to 0 Rule Set:
    {
     name maxujobs
     limit users * to slots=20
    }

    {
     name max_linux
     limit users * hosts @linux to slots=5
    }

    {
     name max_per_host
     limit users roland hosts {@linux} to slots=2
     limit users {*} hosts {@linux} to slots=1
     limit users * hosts * to slots=0
    }
qstat Output:
    $ qstat
    job-ID  prior   name       user         state submit/start at     queue        slots ja-task-ID
    -----------------------------------------------------------------------------------------------
         27 0.55500 Sleeper    roland       r     02/21/2006 15:53:10 all.q@carc    1
         29 0.55500 Sleeper    roland       r     02/21/2006 15:53:10 all.q@carc    1
         30 0.55500 Sleeper    roland       r     02/21/2006 15:53:10 all.q@durin   1
         26 0.55500 Sleeper    roland       r     02/21/2006 15:53:10 all.q@durin   1
         28 0.55500 Sleeper    user1        r     02/21/2006 15:53:10 all.q@durin   1
qquota Output:
    $ qquota # as user roland
    resource quota rule    limit            filter
    --------------------------------------------------------------------------------
    maxujobs/1         slots=5/20           -
    max_linux/1        slots=5/5            hosts @linux
    max_per_host/1     slots=2/2            users roland hosts durin
    max_per_host/1     slots=2/2            users roland hosts carc

    $ qquota -h durin # as user roland
    resource quota    limit                filter
    --------------------------------------------------------------------------------
    maxujobs/1         slots=5/20           -
    max_linux/1        slots=5/5            hosts @linux
    max_per_host/1     slots=2/2            users roland hosts durin

    $ qquota -u user1
    resource quota    limit                filter
    --------------------------------------------------------------------------------
    maxujobs/1         slots=5/20           -
    max_linux/1        slots=5/5            hosts @linux
    max_per_host/1     slots=1/2            users user1 hosts durin

    $ qquota -u *
    resource quota    limit                filter
    --------------------------------------------------------------------------------
    maxujobs/1         slots=5/20           -
    max_linux/1        slots=5/5            hosts @linux
    max_per_host/1     slots=2/2            users roland hosts carc
    max_per_host/1     slots=2/2            users roland hosts durin
    max_per_host/1     slots=1/2            users user1 hosts durin

qquota XML Schema:

    <?xml version="1.0" encoding="UTF-8"?>

    <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                elementFormDefault="qualified">

       <xsd:element name="qquota_result">
          <xsd:sequence>
           <xsd:element name="qquota_rule" type="QQuotaRuleType" minOccurs="0" maxOccurs="unbounded"/>
          </xsd:sequence>
       </xsd:element>

       <xsd:complexType name="QQuotaRuleType">
          <xsd:sequence>
             <xsd:element name="user" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="xuser" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="project" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="xproject" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="pe" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="xpe" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="queue" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="xqueue" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="host" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="xhost" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>
             <xsd:element name="limit" type="ResourceLimitType" minOccurs="0" maxOccurs="unbounded"/>
          </xsd:sequence>
          <xsd:attribute name="name" type="xsd:string" use="required"/>
       </xsd:complexType>

       <xsd:complexType name="ResourceLimitType">
          <xsd:attribute name="resource" type="xsd:string" use="required"/>
          <xsd:attribute name="limit" type="xsd:string" use="required"/>
          <xsd:attribute name="value" type="xsd:string" use="optional"/>
       </xsd:complexType>

    </xsd:schema>

##### Internal data structures:

###### Additional Cull Lists

All lists are used by qmaster and scheduler

File sge_resource_quotaL.h

    #ifndef __SGE_RESOURCE_QUOTAL_H
    #define __SGE_RESOURCE_QUOTAL_H

    #include "sge_boundaries.h"
    #include "cull.h"

    #ifdef  __cplusplus
    extern "C" {
    #endif

    /* *INDENT-OFF* */

    /* Resource Quota Set */
    enum {
       RQS_name = RQS_LOWERBOUND,
       RQS_description,
       RQS_enabled,
       RQS_rule
    };

    LISTDEF(RQS_Type)
     JGDI_ROOT_OBJ(ResourceQuotaSet, SGE_RQS_LIST, ADD | MODIFY | DELETE | GET | GET_LIST)
     JGDI_EVENT_OBJ(ADD(sgeE_RQS_ADD) | MODIFY(sgeE_RQS_MOD) | DELETE(sgeE_RQS_DEL) | GET_LIST(sgeE_RQS_LIST))
     SGE_STRING(RQS_name, CULL_PRIMARY_KEY | CULL_HASH | CULL_UNIQUE | CULL_SPOOL)
     SGE_STRING(RQS_description, CULL_DEFAULT | CULL_SPOOL)
     SGE_BOOL(RQS_enabled, CULL_DEFAULT | CULL_SPOOL)
     SGE_LIST(RQS_rule, RQR_Type, CULL_DEFAULT | CULL_SPOOL)
    LISTEND

    NAMEDEF(RQSN)
       NAME("RQS_name")
       NAME("RQS_description")
       NAME("RQS_enabled")
       NAME("RQS_rule")
    NAMEEND

    #define RQSS sizeof(RQSN)/sizeof(char*)

    /* Resource Quota Rule */
    enum {
       RQR_name = RQR_LOWERBOUND,
       RQR_filter_users,
       RQR_filter_projects,
       RQR_filter_pes,
       RQR_filter_queues,
       RQR_filter_hosts,
       RQR_limit,
       RQR_level
    };

    LISTDEF(RQR_Type)
     JGDI_OBJ(ResourceQuotaRule)
     SGE_STRING(RQR_name, CULL_PRIMARY_KEY | CULL_HASH | CULL_UNIQUE | CULL_SPOOL)
     SGE_OBJECT(RQR_filter_users, RQRF_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_OBJECT(RQR_filter_projects, RQRF_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_OBJECT(RQR_filter_pes, RQRF_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_OBJECT(RQR_filter_queues, RQRF_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_OBJECT(RQR_filter_hosts, RQRF_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_LIST(RQR_limit, RQRL_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_ULONG(RQR_level, CULL_DEFAULT | CULL_JGDI_RO)
    LISTEND

    NAMEDEF(RQRN)
       NAME("RQR_name")
       NAME("RQR_filter_users")
       NAME("RQR_filter_projects")
       NAME("RQR_filter_pes")
       NAME("RQR_filter_queues")
       NAME("RQR_filter_hosts")
       NAME("RQR_limit")
       NAME("RQR_level")
    NAMEEND

    #define RQRS sizeof(RQRN)/sizeof(char*)

    enum {
       FILTER_USERS = 0,
       FILTER_PROJECTS,
       FILTER_PES,
       FILTER_QUEUES,
       FILTER_HOSTS
    };

    enum {
       RQR_ALL = 0,
       RQR_GLOBAL,
       RQR_CQUEUE,
       RQR_HOST,
       RQR_QUEUEI
    };

    /* Resource Quota Rule Filter */
    enum {
       RQRF_expand = RQRF_LOWERBOUND,
       RQRF_scope,
       RQRF_xscope
    };

    LISTDEF(RQRF_Type)
     JGDI_OBJ(ResourceQuotaRuleFilter)
     SGE_BOOL(RQRF_expand, CULL_DEFAULT | CULL_SPOOL)
     SGE_LIST(RQRF_scope, ST_Type, CULL_DEFAULT | CULL_SPOOL)
     SGE_LIST(RQRF_xscope, ST_Type, CULL_DEFAULT | CULL_SPOOL)
    LISTEND

    NAMEDEF(RQRFN)
       NAME("RQRF_expand")
       NAME("RQRF_scope")
       NAME("RQRF_xscope")
    NAMEEND

    #define RQRFS sizeof(RQRFN)/sizeof(char*)

    /* Resource Quota Rule Limit */
    enum {
       RQRL_name = RQRL_LOWERBOUND,
       RQRL_value,
       RQRL_type,
       RQRL_dvalue,
       RQRL_usage,
       RQRL_dynamic
    };

    LISTDEF(RQRL_Type)
     JGDI_OBJ(ResourceQuotaRuleLimit)
     SGE_STRING(RQRL_name, CULL_PRIMARY_KEY | CULL_HASH | CULL_UNIQUE | CULL_SPOOL)
     SGE_STRING(RQRL_value, CULL_DEFAULT | CULL_SPOOL)
     SGE_ULONG(RQRL_type, CULL_DEFAULT | CULL_SPOOL | CULL_JGDI_RO)
     SGE_DOUBLE(RQRL_dvalue, CULL_DEFAULT | CULL_SPOOL | CULL_JGDI_RO)
     SGE_LIST(RQRL_usage, RUE_Type, CULL_DEFAULT | CULL_JGDI_RO)
     SGE_BOOL(RQRL_dynamic, CULL_DEFAULT | CULL_JGDI_RO)
    LISTEND

    NAMEDEF(RQRLN)
       NAME("RQRL_name")
       NAME("RQRL_value")
       NAME("RQRL_type")
       NAME("RQRL_dvalue")
       NAME("RQRL_usage")
       NAME("RQRL_dynamic")
    NAMEEND

    #define RQRLS sizeof(RQRLN)/sizeof(char*)

    /* *INDENT-ON* */

    #ifdef  __cplusplus
    }
    #endif
    #endif /* __SGE_RESOURCE_QUOTAL_H */

###### Additional GDI requests

- SGE_GDI_ADD(RQS, resource_quota)

This request allows for adding a new resource quota set. It contains the complete rule set configuration and is used for implementing the qconf option '-arqs' and '-Arqs'.

- SGE_GDI_MOD(RQS, resource_quota)

This request allows for changing the complete resource quota set configuration. It contains a full rule set configuration and is used for implementing qconf option '-mrqs' and '-Mrqs'.

- SGE_GDI_DEL(RQS, resource_quota)

This request allows for removing a complete resource quota set configuration. It contains only the name of the resource quota to be removed and is used for implementing the qconf option '-drqs'.

- SGE_GDI_GET(RQS,where,what)

This request allows for retrieving resource quota sets. CULL 'where' expressions can be used for selecting particular rule sets, CULL 'what' expressions can be used for selecting particular rule set fields. The SGE_GDI_GET request is used for implementing the qconf option '-srqs'.

- SGE_GDI_MOD(LISR, resource_quota, fields)
- SGE_GDI_MOD(LISR, resource_quota, fields) + SGE_GDI_SET()

These requests are a SGE_GDI_MOD(LISR, resource_quota) variation and allow for replacing the selected fields within a resource quota. Field selection is done by means of an incomplete resource quota set configuration structure. The requests are used for implementing qconf options '-rattr' and '-Rattr'.

- SGE_GDI_MOD(LISR, resource_quota, fields) + SGE_GDI_APPEND(rule_identifiers, list_elements)

This request allows for adding one or more list elements regarding to one or more rule identifiers to each of the selected list fields within a resource quota set configuration. Field selections are done by means of an incomplete rule set configuration structure. The rule_identifiers of each tuple below each selected rule set field are used to define which rule should be modified. All list elements belonging to each tuple are added. Already existing list elements are silently overwritten, also if the selected rule configuration is not a list field this silently overwrites the current setting.The request is for implementing the qconf option '-aatrr' and '-Aattr'.

- SGE_GDI_MOD(LISR, resource_quota, fields) + SGE_GDI_CHANGE(rule_identifiers, list_elements)

This request allows for replacing one or more list elements regarding to one or more rule identifiers to each of the selected list fields within a resource quota set configuration. Field selections are done by means of an incomplete rule set configuration structure. The request is for implementing the qconf option -mattr' and '-Mattr'

- SGE_GDI_MOD(RQS, resource_quota, fields) + SGE_GDI_REMOVE(rule_identifiers, list_elements)

This request allows for removing one or more list elements regarding to one or more rule identifiers to each of the selected list fields within a resource quota set configuration. Field selections are done by means of an incomplete rule set configuration structure. The request is for implementing the qconf option -dattr' and '-dattr'

###### Additional Event Client requests

- sgeE_RQS_LIST

This event is sent once directly after event client registration to initialize the resource quota set list and contains the complete list of all resource quota sets with all configuration.

- sgeE_RQS_ADD(resource_quota)

This event is sent each time when a new resource quota set configuration has been created. It contains the full resource quota set configuration, but no usage information.

- sgeE_RQS_DEL(resource_quota)

This event is sent each time when an existing resource quota set configuration is removed and contains only the name of the resource quota to be removed.

- sgeE_RQS_MOD(resource_quota)

This event is sent each time when an existing resource quota set configuration changes. It contains the full resource quota set configuration.

- sgeE_RQS_ADD(resource_quota, rule_identifier, usage)
- sgeE_RQS_MOD(resource_quota, rule_identifier, usage)
- sgeE_RQS_DEL(resource_quota, rule_identifier, usage)

These events are send each time when a usage object was added, modified or deleted. The resource_quota and rule_identifier contains only the name of the object to be edited. The usage object is the object to be modified.

###### Qmaster additions:

- add cull rule set definition (Internal data structures)
- spooling code for the rule sets (mainly for classic spooling)
- update resource usage in all rules

###### Scheduler additions:

- Create Resource Reservation Structure
  - prepare_resource_schedules()
- Scheduling matching code
  - sge_sequential_assignment()
  - sge_select_parallel_environment()
- Debit Code
  - debit_scheduled_job()

###### lib additions:

- add code for book keeping of resource usage

###### Book keeping of usage:

- started jobs
- finished/deleted/running jobs
- (suspended jobs)
- object modify (queue host)
- object add/delete (queue host)