Concepts

Nimrod/G Concepts

Types

Identifier

The basic type used throughout Nimrod/G for names and other, you guessed it, identifiers.

Format

^[a-zA-Z0-9_]+$

Examples

• experiment1
• my_awesome_experiment

Configuration Key

Used to uniquely identify a global configuration entry. These are essentially identifiers joined with a period (.).

Format

^[a-zA-Z0-9_]+(?:(?:\\.[a-zA-Z0-9_]+)+)?$

Examples

• nimrod.workdir
• nimrod.master.poll_interval
• nimrod.serve.port
• nimrod.master.amqp.uri

Resource Path

Used to uniquely identify a compute resource. These are essentially identifiers joined with a forwardslash (/).

Format

^[a-zA-Z0-9_]+(?:(?:/[a-zA-Z0-9_]+)+)?$

Examples

• tinaroo/intel
• nectar/intel/4gb/0

Variable Name

The name of a run variable. This follows exactly the same rules as C does for its identifiers.

Format

^[a-zA-Z_](?:[a-zA-Z0-9_])*$

Examples

• a
• b
• _a
• a0

Experiment

Experiments are a top-level primitive. They are a logical grouping of runs.

Each experiment has its own working directory and is identified by a unique name, which is a Identifier.

Experiments are logical groupings of jobs.

Job

Variable

Tasks

Commands

A command is just that, a thing that commands the Nimrod/G agent to do something.

There are three types of commands that Nimrod/G understands: onerror, copy, and exec.

:WARNING: These are NOT the same as the commands listed in .:formats:task-format. The task commands are compiled into these commands.

• onerror commands instruct the agent about what to do if a subsequent command fails.
• redirect commands configure stdout/stderr redirection for subsequent commands.
• copy commands copy a file.
• exec commands execute a program or the shell.

Each command has two different forms, normalised, and denormalised.

• The denormalised form of a command is a human-friendly representation containing the fields unique to each command type.
• The normalised form of a command is the more computer-friendly representation that contains a flat list of arguments. Depending of the type of the command, additional constraints may be be placed on the argument list.

The Nimrod API uses denormalised for most of its operations.

:NOTE: The PostgreSQL backend for Nimrod/G stores its commands internally in normalised form.

:TODO: Currently the wire-format for an agent payload has the commands in denormalised form. This should be changed to normalised form to simplify the (de)serialisation code.

onerror

Instruct the agent about what to do if a subsequent command fails.

Normalised Form

Index	Argument	Allowed Values	Substitutions Allowed?
0	action	`^(fail	ignore)$`

Denormalised JSON

{
	"type": "onerror",
	"action": "fail"
}

redirect

Normalised Form

Tell the agent what to do with subsequent command's stdout/stderr.

Index	Argument	Allowed Values	Substitutions Allowed?
0	stream	`^(stdout	stderr)$`
1	append	`^(true	false)$`
2	file	-	Yes

• If file is an empty string, the stream is ignored.

Denormalised JSON

{
	"type": "redirect",
	"stream": "stdout",
	"append": false,
	"file": {
		"text": "/dev/null",
		"substitutions": []
	}
}

copy

Command the agent to copy a file.

• source_context The context where the source file exists. It should either be root, or node.
• source_path A UNIX-style path representing the location of the file on the source context.
• destination_context The context where the source file should be copied to. It should either be root, or node.
• destination_path A UNIX-style path representing the destination of the file on the destination context.

Normalised Form

Index	Argument	Allowed Values	Substitutions Allowed?
0	source_context	`^(root	node)$`
1	source_path	-	Yes
2	destination_context	`^(root	node)$`
3	destination_path	-	Yes

Denormalised JSON

{
	"type": "copy",
	"source_context": "node",
	"source_path": {
		"text": "/dev/null",
		"substitutions": []
	},
	"destination_context": "root",
	"destination_path": {
		"text": "/dev/null",
		"substitutions": []
	}
}

exec

Command the agent to copy a file.

• search_path When searching for the program, should the system's PATH be used? It should either be true, or false.
• program A UNIX-style path representing the location of the program/script to execute.
  • If this is an empty string, this is assumed to be a shell command.
• arguments A list of arguments to be passed to program, including ''argv[0]''

Remarks

• If program is empty, arguments is expected to be of size 1 and to contain the command line to pass to the shell. search_path should be false.

Normalised Form

Index	Argument	Allowed Values	Substitutions Allowed?
0	search_path	`^(true	false)$`
1	program	-	No
2	arguments	-	Yes

Denormalised JSON

{
	"type": "exec",
	"program": "python",
	"search_path": true,
	"arguments": [
		{
			"text": "python",
			"substitutions": []
		},
		{
			"text": "script.py",
			"substitutions": []
		}
	]
}

Command Arguments

Command arguments represent a string of text that may contain substitutions.

These are little more than a container for a string and a list of substitution definitions.

Substitutions

Substitutions contain three fields:

• name - The name of the variable being substituted.
• start_index - Given the original un-substituted string, the index where the substitution starts.
• end_index - Given the original un-substituted string, the index where the substitution ends.
• relative_start_index - Relative to the end of the previous substitution, the index at which this substitution starts.

Examples

Example 1

Argument:

"START: $x, ${y}"

Substitutions:

Index	Name	Start	End	Relative
0	x	7	9	7
1	y	11	15	2

Example 2

Argument:

"$x, ${y} :END"

Substitutions:

Index	Name	Start	End	Relative
0	x	0	2	0
1	y	4	8	2

Example 3

Argument:

"START: $x, ${y} :END"

Substitutions:

Index	Name	Start	End	Relative
0	x	7	9	7
1	y	11	15	2

Resource

A resource is a logical representation of a endpoint that is capable of running jobs.

Resources are stored as a tree.

Actuator

actuate - /ˈæktʃu.eɪt/ - To activate, or to put into motion; to animate.

In layman's terms, they make it go.

An actuator controls a top-level (root) resource, and any of its children.

• An actuator's job is to:
  • Open a channel to the resource allowing the Master to schedule Agents on it.
  • Spawn agents on a given resource node.
    • An actuator is only responsible for its owned root resource and any of its children. Agent spawning is expected to fail if the requested resource node is from a different root.
  • Clean up the resource upon termination.
    • Although the agents will attempt to clean as part of their exit process, this may not be possible due to premature forced termination. It is expected that the actuator attempt to clean any mess left behind.
• Actuators do NOT run on the resource itself, they run as part of the master process and are expected to remotely control the resource (such as via SSH).

Agent

Master

Scribbles

Various whiteboard scribbles.

Experiment Concepts