API Docs

Security

class skein.Security

Security configuration.

Parameters:
cert_path : str

Path to the certificate file in pem format.

key_path : str

Path to the key file in pem format.

classmethod from_default()

The default security configuration.

classmethod from_directory(directory)

Create a security object from a directory.

Relies on standard names for each file (skein.crt and skein.pem).

classmethod from_new_directory(directory=None, force=False)

Create a Security object from a new certificate/key pair.

This is equivalent to the cli command skein config gencerts with the option to specify an alternate directory if needed. Should only need to be called once per user upon install. Call again with force=True to generate new TLS keys and certificates.

Parameters:
directory : str, optional

The directory to write the configuration to. Defaults to the global skein configuration directory at ~/.skein/.

force : bool, optional

If True, will overwrite existing configuration. Otherwise will error if already configured. Default is False.

Client

class skein.Client(address=None, security=None, log=None)

Connect to and schedule applications on the YARN cluster.

Parameters:
address : str, optional

The address for the daemon. By default will create a new daemon process. Pass in address explicitly to connect to a different daemon. To connect to the global daemon see Client.from_global_daemon.

security : Security, optional

The security configuration to use to communicate with the daemon. Defaults to the global configuration.

log : str, bool, or None, optional

When starting a new daemon, sets the logging behavior for the daemon. Values may be a path for logs to be written to, None to log to stdout/stderr, or False to turn off logging completely. Default is None.

Examples

>>> with skein.Client() as client:
...     app_id = client.submit('spec.yaml')
application_report(app_id)

Get a report on the status of a skein application.

Parameters:
app_id : str

The id of the application.

Returns:
report : ApplicationReport

Examples

>>> client.application_report('application_1526134340424_0012')
ApplicationReport<name='demo'>
close()

Closes the java daemon if started by this client. No-op otherwise.

connect(app_id, wait=True)

Connect to a running application.

Parameters:
app_id : str

The id of the application.

wait : bool, optional

If true [default], blocks until the application starts. If False, will raise a ApplicationNotRunningError immediately if the application isn’t running.

Returns:
app_client : ApplicationClient
Raises:
ApplicationNotRunningError

If the application isn’t running.

classmethod from_global_daemon()

Connect to the global daemon.

get_applications(states=None)

Get the status of current skein applications.

Parameters:
states : sequence of ApplicationState, optional

If provided, applications will be filtered to these application states. Default is ['SUBMITTED', 'ACCEPTED', 'RUNNING'].

Returns:
reports : list of ApplicationReport

Examples

Get all the finished and failed applications

>>> client.get_applications(states=['FINISHED', 'FAILED'])
[ApplicationReport<name='demo'>,
 ApplicationReport<name='dask'>,
 ApplicationReport<name='demo'>]
kill_application(app_id)

Kill an application.

Parameters:
app_id : str

The id of the application to kill.

static start_global_daemon(log=None)

Start the global daemon.

No-op if the global daemon is already running.

Parameters:
log : str, bool, or None, optional

Sets the logging behavior for the daemon. Values may be a path for logs to be written to, None to log to stdout/stderr, or False to turn off logging completely. Default is None.

Returns:
address : str

The address of the daemon

static stop_global_daemon()

Stops the global daemon if running.

No-op if no global daemon is running.

submit(spec)

Submit a new skein application.

Parameters:
spec : ApplicationSpec, str, or dict

A description of the application to run. Can be an ApplicationSpec object, a path to a yaml/json file, or a dictionary description of an application specification.

Returns:
app_id : str

The id of the submitted application.

submit_and_connect(spec)

Submit a new skein application, and wait to connect to it.

If an error occurs before the application connects, the application is killed.

Parameters:
spec : ApplicationSpec, str, or dict

A description of the application to run. Can be an ApplicationSpec object, a path to a yaml/json file, or a dictionary description of an application specification.

Returns:
app_client : ApplicationClient

Application Client

class skein.ApplicationClient(address, app_id, security=None)

A client for the application master.

Used to interact with a running application.

Parameters:
address : str

The address of the application master.

app_id : str

The application id

security : Security, optional

The security configuration to use to communicate with the daemon. Defaults to the global configuration.

classmethod from_current()

Create an application client from within a running container.

Useful for connecting to the application master from a running container in a application.

get_containers(services=None, states=None)

Get information on containers in this application.

Parameters:
services : sequence of str, optional

If provided, containers will be filtered to these services. Default is all services.

states : sequence of ContainerState, optional

If provided, containers will be filtered by these container states. Default is ['WAITING', 'REQUESTED', 'RUNNING'].

Returns:
containers : list of Container
get_specification()

Get the specification for the running application.

Returns:
spec : ApplicationSpec
kill_container(id)

Kill a container.

Parameters:
id : str

The id of the container to kill.

kv

The Skein Key-Value store.

Used by applications to coordinate configuration and global state.

This implements the standard MutableMapping interface, along with the ability to “wait” for keys to be set.

Examples

>>> app_client.kv['foo'] = 'bar'
>>> app_client.kv['foo']
'bar'
>>> del app_client.kv['foo']
>>> 'foo' in app_client.kv
False

Wait until the key is set, either by another service or by a user client. This is useful for inter-service synchronization.

>>> app_client.kv.wait('mykey')
scale(service, instances)

Scale a service to a requested number of instances.

Adds or removes containers to match the requested number of instances. When choosing which containers to remove, containers are removed in order of state (WAITING, REQUESTED, RUNNING) followed by age (oldest to newest).

Parameters:
service : str, optional

The service to scale.

instances : int

The number of instances to scale to.

Returns:
containers : list of Container

A list of containers that were started or stopped.

shutdown(status='SUCCEEDED')

Shutdown the application.

Stop all running containers and shutdown the application.

Parameters:
status : FinalStatus, optional

The final application status. Default is ‘SUCCEEDED’.

Key Value Store

class skein.core.KeyValueStore(client)

The Skein Key-Value store.

Used by applications to coordinate configuration and global state.

This implements the standard MutableMapping interface, along with the ability to “wait” for keys to be set.

to_dict()

Return the whole key-value store as a dictionary

wait(key)

Get a key from the key-value store, blocking until the key is set.

Application Specification

class skein.ApplicationSpec(services=required, name='skein', queue='default', tags=None, max_attempts=1)

A complete description of an application.

Parameters:
services : dict

A mapping of service-name to services. At least one service is required.

name : string, optional

The name of the application, defaults to ‘skein’.

queue : string, optional

The queue to submit to. Defaults to the default queue.

tags : set, optional

A set of strings to use as tags for this application.

max_attempts : int, optional

The maximum number of submission attempts before marking the application as failed. Note that this only considers failures of the application master during startup. Default is 1.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_file(path, format='infer')

Create an instance from a json or yaml file.

Parameters:
path : str

The path to the file to load.

format : {‘infer’, ‘json’, ‘yaml’}, optional

The file format. By default the format is inferred from the file extension.

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_file(path, format='infer', skip_nulls=True)

Write object to a file.

Parameters:
path : str

The path to the file to load.

format : {‘infer’, ‘json’, ‘yaml’}, optional

The file format. By default the format is inferred from the file extension.

skip_nulls : bool, optional

By default null values are skipped in the output. Set to True to output all fields.

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.Service(commands=required, resources=required, instances=1, max_restarts=0, files=None, env=None, depends=None)

Description of a Skein service.

Parameters:
commands : list

Shell commands to startup the service. Commands are run in the order provided, with subsequent commands only run if the prior commands succeeded. At least one command must be provided

resources : Resources

Describes the resources needed to run the service.

instances : int, optional

The number of instances to create on startup. Default is 1.

max_restarts : int, optional

The maximum number of restarts to allow for this service. Containers are only restarted on failure, and the cap is set for all containers in the service, not per container. Set to -1 to allow infinite restarts. Default is 0.

files : dict, optional

Describes any files needed to run the service. A mapping of destination relative paths to File or str objects describing the sources for these paths. If a str, the file type is inferred from the extension.

env : dict, optional

A mapping of environment variables needed to run the service.

depends : set, optional

A set of service names that this service depends on. The service will only be started after all its dependencies have been started.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.FileType

Enum of possible file types to distribute with the application.

Attributes:
FILE : FileType

Regular file

ARCHIVE : FileType

A .zip, .tar.gz, or .tgz file to be automatically unarchived in the containers.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.FileVisibility

Enum of possible file visibilities.

Determines how the file can be shared between containers.

Attributes:
APPLICATION : FileVisibility

Shared only among containers of the same application on the node.

PUBLIC : FileVisibility

Shared by all users on the node.

PRIVATE : FileVisibility

Shared among all applications of the same user on the node.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.File(source=required, type='infer', visibility=FileVisibility.APPLICATION, size=0, timestamp=0)

A file/archive to distribute with the service.

Parameters:
source : str

The path to the file/archive. If no scheme is specified, path is assumed to be on the local filesystem (file:// scheme).

type : FileType or str, optional

The type of file to distribute. Archive’s are automatically extracted by yarn into a directory with the same name as their destination. By default the type is inferred from the file extension.

visibility : FileVisibility or str, optional

The resource visibility, default is FileVisibility.APPLICATION

size : int, optional

The resource size in bytes. If not provided will be determined by the file system.

timestamp : int, optional

The time the resource was last modified. If not provided will be determined by the file system.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.Resources(memory=required, vcores=required)

Resource requests per container.

Parameters:
memory : int

The amount of memory to request, in MB. Requests smaller than the minimum allocation will receive the minimum allocation (usually 1024). Requests larger than the maximum allocation will error on application submission.

vcores : int

The number of virtual cores to request. Depending on your system configuration one virtual core may map to a single actual core, or a fraction of a core. Requests larger than the maximum allocation will error on application submission.

classmethod from_dict(obj)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(msg)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

Application Responses

class skein.model.ApplicationState

Enum of application states.

Attributes:
NEW : ApplicationState

Application was just created.

NEW_SAVING : ApplicationState

Application is being saved.

SUBMITTED : ApplicationState

Application has been submitted.

ACCEPTED : ApplicationState

Application has been accepted by the scheduler.

RUNNING : ApplicationState

Application is currently running.

FINISHED : ApplicationState

Application finished successfully.

FAILED : ApplicationState

Application failed.

KILLED : ApplicationState

Application was terminated by a user or admin.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.model.FinalStatus

Enum of application final statuses.

Attributes:
SUCCEEDED : FinalStatus

Application finished successfully.

KILLED : FinalStatus

Application was terminated by a user or admin.

FAILED : FinalStatus

Application failed.

UNDEFINED : FinalStatus

Application has not yet finished.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.model.ApplicationReport(id, name, user, queue, tags, host, port, tracking_url, state, final_status, progress, usage, diagnostics, start_time, finish_time)

Report of application status.

Parameters:
id : str

The application ID.

name : str

The application name.

user : str

The user that started the application.

queue : str

The application queue.

tags : set of strings

The application tags.

host : str

The host the application master is running on.

port : int

The rpc port for the application master

tracking_url : str

The application tracking url.

state : ApplicationState

The application state.

final_status : FinalStatus

The application final status.

progress : float

The progress of the application, from 0.0 to 1.0.

usage : ResourceUsageReport

Report on application resource usage.

diagnostics : str

The diagnostic message in the case of failures.

start_time : datetime

The application start time.

finish_time : datetime

The application finish time.

classmethod from_dict(obj)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

runtime

The total runtime of the container.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.model.ResourceUsageReport(memory_seconds, vcore_seconds, num_used_containers, needed_resources, reserved_resources, used_resources)

Resource usage report.

Parameters:
memory_seconds : int

The total amount of memory (in MBs) the application has allocated times the number of seconds the application has been running.

vcore_seconds : int

The total number of vcores that the application has allocated times the number of seconds the application has been running.

num_used_containers : int

Current number of containers in use.

needed_resources : Resources

The needed resources.

reserved_resources : Resources

The reserved resources.

used_resources : Resources

The used resources.

classmethod from_dict(obj)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

Exceptions

exception skein.SkeinError

Bases: Exception

Base class for Skein specific exceptions

exception skein.SkeinConfigurationError

Bases: skein.exceptions.SkeinError, FileNotFoundError

Skein configuration was not found

exception skein.ConnectionError

Bases: skein.exceptions.SkeinError, ConnectionError

Failed to connect to the daemon or application master

exception skein.DaemonNotRunningError

Bases: skein.exceptions.ConnectionError

The daemon process is not currently running

exception skein.ApplicationNotRunningError

Bases: skein.exceptions.ConnectionError

The application master is not currently running

exception skein.DaemonError

Bases: skein.exceptions.SkeinError

Internal exceptions from the daemon

exception skein.ApplicationError

Bases: skein.exceptions.SkeinError

Internal exceptions from the application master