API Docs

Security

class skein.Security

Security configuration.

Parameters:
cert_path : str

Path to the certificate file in pem format.

key_path : str

Path to the key file in pem format.

classmethod from_default()

The default security configuration.

classmethod from_directory(directory)

Create a security object from a directory.

Relies on standard names for each file (skein.crt and skein.pem).

classmethod from_new_directory(directory=None, force=False)

Create a Security object from a new certificate/key pair.

This is equivalent to the cli command skein config gencerts with the option to specify an alternate directory if needed. Should only need to be called once per user upon install. Call again with force=True to generate new TLS keys and certificates.

Parameters:
directory : str, optional

The directory to write the configuration to. Defaults to the global skein configuration directory at ~/.skein/.

force : bool, optional

If True, will overwrite existing configuration. Otherwise will error if already configured. Default is False.

Client

class skein.Client(address=None, security=None, log=None)

Connect to and schedule applications on the YARN cluster.

Parameters:
address : str, optional

The address for the daemon. By default will create a new daemon process. Pass in address explicitly to connect to a different daemon. To connect to the global daemon see Client.from_global_daemon.

security : Security, optional

The security configuration to use to communicate with the daemon. Defaults to the global configuration.

log : str, bool, or None, optional

When starting a new daemon, sets the logging behavior for the daemon. Values may be a path for logs to be written to, None to log to stdout/stderr, or False to turn off logging completely. Default is None.

Examples

>>> with skein.Client() as client:
...     app_id = client.submit('spec.yaml')
application_report(app_id)

Get a report on the status of a skein application.

Parameters:
app_id : str

The id of the application.

Returns:
report : ApplicationReport

Examples

>>> client.application_report('application_1526134340424_0012')
ApplicationReport<name='demo'>
close()

Closes the java daemon if started by this client. No-op otherwise.

connect(app_id, wait=True)

Connect to a running application.

Parameters:
app_id : str

The id of the application.

wait : bool, optional

If true [default], blocks until the application starts. If False, will raise a ApplicationNotRunningError immediately if the application isn’t running.

Returns:
app_client : ApplicationClient
Raises:
ApplicationNotRunningError

If the application isn’t running.

classmethod from_global_daemon()

Connect to the global daemon.

get_applications(states=None)

Get the status of current skein applications.

Parameters:
states : sequence of ApplicationState, optional

If provided, applications will be filtered to these application states. Default is ['SUBMITTED', 'ACCEPTED', 'RUNNING'].

Returns:
reports : list of ApplicationReport

Examples

Get all the finished and failed applications

>>> client.get_applications(states=['FINISHED', 'FAILED'])
[ApplicationReport<name='demo'>,
 ApplicationReport<name='dask'>,
 ApplicationReport<name='demo'>]
kill_application(app_id)

Kill an application.

Parameters:
app_id : str

The id of the application to kill.

static start_global_daemon(log=None)

Start the global daemon.

No-op if the global daemon is already running.

Parameters:
log : str, bool, or None, optional

Sets the logging behavior for the daemon. Values may be a path for logs to be written to, None to log to stdout/stderr, or False to turn off logging completely. Default is None.

Returns:
address : str

The address of the daemon

static stop_global_daemon()

Stops the global daemon if running.

No-op if no global daemon is running.

submit(spec)

Submit a new skein application.

Parameters:
spec : ApplicationSpec, str, or dict

A description of the application to run. Can be an ApplicationSpec object, a path to a yaml/json file, or a dictionary description of an application specification.

Returns:
app_id : str

The id of the submitted application.

submit_and_connect(spec)

Submit a new skein application, and wait to connect to it.

If an error occurs before the application connects, the application is killed.

Parameters:
spec : ApplicationSpec, str, or dict

A description of the application to run. Can be an ApplicationSpec object, a path to a yaml/json file, or a dictionary description of an application specification.

Returns:
app_client : ApplicationClient

Application Client

class skein.ApplicationClient(address, app_id, security=None)

A client for the application master.

Used to interact with a running application.

Parameters:
address : str

The address of the application master.

app_id : str

The application id

security : Security, optional

The security configuration to use to communicate with the daemon. Defaults to the global configuration.

classmethod from_current()

Create an application client from within a running container.

Useful for connecting to the application master from a running container in a application.

get_containers(services=None, states=None)

Get information on containers in this application.

Parameters:
services : sequence of str, optional

If provided, containers will be filtered to these services. Default is all services.

states : sequence of ContainerState, optional

If provided, containers will be filtered by these container states. Default is ['WAITING', 'REQUESTED', 'RUNNING'].

Returns:
containers : list of Container
get_specification()

Get the specification for the running application.

Returns:
spec : ApplicationSpec
kill_container(id)

Kill a container.

Parameters:
id : str

The id of the container to kill.

kv

The Skein Key-Value store.

Used by applications to coordinate configuration and global state.

This implements the standard MutableMapping interface, along with the ability to “wait” for keys to be set. Keys are strings, with values as bytes.

Examples

>>> app_client.kv['foo'] = b'bar'
>>> app_client.kv['foo']
b'bar'
>>> del app_client.kv['foo']
>>> 'foo' in app_client.kv
False

Wait until the key is set, either by another service or by a user client. This is useful for inter-service synchronization.

>>> app_client.kv.wait('mykey')
scale(service, instances)

Scale a service to a requested number of instances.

Adds or removes containers to match the requested number of instances. When choosing which containers to remove, containers are removed in order of state (WAITING, REQUESTED, RUNNING) followed by age (oldest to newest).

Parameters:
service : str, optional

The service to scale.

instances : int

The number of instances to scale to.

Returns:
containers : list of Container

A list of containers that were started or stopped.

shutdown(status='SUCCEEDED')

Shutdown the application.

Stop all running containers and shutdown the application.

Parameters:
status : FinalStatus, optional

The final application status. Default is ‘SUCCEEDED’.

Runtime Properties

skein.properties = <skein.core.Properties object>

Skein runtime properties.

This class implements an immutable mapping type, exposing properties determined at import time.

Attributes:
application_id : str or None

The current application id. None if not running in a container.

appmaster_address : str or None

The address of the current application’s appmaster. None if not running in a container.

config_dir : str

The path to the configuration directory.

container_id : str or None

The current skein container id (of the form '{service}_{instance}'). None if not running in a container.

container_resources : Resources or None

The resources allocated to the current container. None if not in a container.

yarn_container_id : str or None

The current YARN container id. None if not running in a container.

Key Value Store

class skein.kv.KeyValueStore(client)

The Skein Key-Value store.

Used by applications to coordinate configuration and global state.

clear() → None. Remove all items from D.
count(end=None, prefix=None)

Count keys in the key-value store.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

prefix : str, optional

If provided, will count the number keys matching this prefix.

Returns:
int
discard()

Discard a single key.

Returns true if the key was present, false otherwise.

Parameters:
key : str

The key to discard.

Returns:
bool
discard_prefix(return_keys=False)

Discard all key-value pairs whose keys start with prefix.

Returns either the number of keys discarded or a list of those keys, depending on the value of return_keys.

Parameters:
prefix : str

The key prefix.

return_keys : bool, optional

If True, the discarded keys will be returned instead of their count. Default is False.

Returns:
int or list of keys
discard_range(end=None, return_keys=False)

Discard a range of keys.

Returns either the number of keys discarded or a list of those keys, depending on the value of return_keys.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_keys : bool, optional

If True, the discarded keys will be returned instead of their count. Default is False.

Returns:
int or list of keys
event_queue()

Create a new EventQueue subscribed to no events.

Examples

Subscribe to events starting with 'foo' or 'bar'.

>>> foo = skein.kv.EventFilter(prefix='foo')
>>> bar = skein.kv.EventFilter(prefix='bar')
>>> queue = app.kv.event_queue()              # doctest: skip
>>> queue.subscribe(foo)                      # doctest: skip
>>> queue.subscribe(bar)                      # doctest: skip
>>> for event in queue:                       # doctest: skip
...     if event.filter == foo:
...         print("foo event")
...     else:
...         print("bar event")
events(event_filter=None, key=None, prefix=None, start=None, end=None, event_type=None)

Shorthand for creating an EventQueue and adding a single filter.

May provide either an explicit event filter, or provide arguments to create a new one and add it to the queue.

If no arguments are provided, creates a queue subscribed to all events.

Parameters:
event_filter : EventFilter

An explicit EventFilter. If provided, no other keyword arguments may be provided.

key : str, optional

If present, only events from this key will be selected.

prefix : str, optional

If present, only events with this key prefix will be selected.

start : str, optional

If present, specifies the lower bound of the key range, inclusive.

end : str, optional

If present, specifies the upper bound of the key range, exclusive.

event_type : EventType, optional.

The type of event. Default is 'ALL'.

Returns:
EventQueue

Examples

Subscribe to all events with prefix 'foo':

>>> for event in app.kv.events(prefix='foo'):  # doctest: skip
...     if event.type == 'PUT':
...         print("PUT<key=%r, value=%r>" % (event.key, event.value))
...     else:  # DELETE
...         print("DELETE<key=%r>" % event.key)
PUT<key='foo', value=b'bar'>
PUT<key='food', value=b'biz'>
DELETE<key='food'>
PUT<key='foo', value=b'changed'>
exists()

Check if a key exists in the key-value store.

Parameters:
key : str

The key to check the presence of.

Returns:
bool
get(default=None, return_owner=False)

Get the value associated with a single key.

Parameters:
key : str

The key to get.

default : bytes or None, optional

Default value to return if the key is not present.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
bytes or ValueOwnerPair
get_prefix(return_owner=False)

Get all key-value pairs whose keys start with prefix.

Parameters:
prefix : str

The key prefix.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
OrderedDict
get_range(end=None, return_owner=False)

Get a range of keys.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
OrderedDict
items() → a set-like object providing a view on D's items
keys() → a set-like object providing a view on D's keys
list_keys(end=None, prefix=None)

Get a list of keys in the key-value store.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

prefix : str, optional

If provided, will return all keys matching this prefix.

Returns:
list of keys
missing()

Check if a key is not in the key-value store.

This is the inverse of exists.

Parameters:
key : str

The key to check the absence of.

Returns:
bool
pop(default=None, return_owner=False)

Remove a single key and return its corresponding value.

Parameters:
key : str

The key to pop.

default : bytes or None, optional

Default value to return if the key is not present.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
bytes or ValueOwnerPair
pop_prefix(return_owner=False)

Remove all key-value pairs whose keys start with prefix, and return their corresponding values.

Parameters:
prefix : str

The key prefix.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
OrderedDict
pop_range(end=None, return_owner=False)

Remove a range of keys and return their corresponding values.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
OrderedDict
popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

put(value=no_change, owner=no_change)

Assign a value and/or owner for a single key.

Parameters:
key : str

The key to put.

value : bytes, optional

The value to put. Default is to leave value unchanged; an error will be raised if the key doesn’t exist.

owner : str or None, optional

The container id to claim ownership. Provide None to set to no owner. Default is to leave value unchanged.

setdefault(key, default)

Get the value associated with key, setting it to default if not present.

This transaction happens atomically on the key-value store.

Parameters:
key : str

The key

default : bytes

The default value to set if the key isn’t present.

Returns:
value : bytes
swap(value=no_change, owner=no_change, return_owner=False)

Assign a new value and/or owner for a single key, and return the previous value.

Parameters:
key : str

The key to put.

value : bytes, optional

The value to put. Default is to leave value unchanged; an error will be raised if the key doesn’t exist.

owner : str or None, optional

The container id to claim ownership. Provide None to set to no owner. Default is to leave value unchanged.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
bytes or ValueOwnerPair
transaction(conditions=None, on_success=None, on_failure=None)

An atomic transaction on the key-value store.

Parameters:
conditions : Condition or sequence of Conditions, optional

A sequence of conditions to evaluate together. The conditional expression succeeds if all conditions evaluate to True, and fails otherwise. If no conditions are provided the conditional expression also succeeds.

on_success : Operation or sequence of Operation, optional

A sequence of operations to apply if all conditions evaluate to True.

on_failure : Operation or sequence of Operation, optional

A sequence of operations to apply if any condition evaluates to False.

Returns:
result : TransactionResult

A namedtuple of (succeeded, results), where results is a list of results from either the on_success or on_failure operations, depending on which branch was evaluated.

Examples

This implements an atomic compare-and-swap operation, a useful concurrency primitive. It sets key to new only if it currently is prev:

>>> from skein import kv
>>> def compare_and_swap(app, key, new, prev):
...     result = app.kv.transaction(
...         conditions=[kv.value(key) == prev],  # if key == prev
...         on_success=[kv.put(key, new)])       # then set key = new
...     return result.succeeded
>>> app.kv['key'] = b'value'  # doctest: skip

Since 'key' currently is b'value', the conditional expression succeeds and 'key' is set to b'new_value'

>>> compare_and_swap(app, 'key', b'new_value', b'value')  # doctest: skip
True

Since 'key' currently is b'value' and not b'wrong', the conditional expression fails and 'key' remains unchanged.

>>> compare_and_swap(app, 'key', b'another_value', b'wrong')  # doctest: skip
False
update(*args, **kwargs)

Update the key-value store with multiple key-value pairs atomically.

Parameters:
arg : mapping or iterable, optional

Either a mapping or an iterable of (key, value).

**kwargs

Extra key-value pairs to set. Semantically these are applied after any present in arg, and will thus override any intersecting keys between the two.

values() → an object providing a view on D's values
wait(key, return_owner=False)

Get the value associated with a single key, blocking until the key exists if not present.

Parameters:
key : str

The key to get.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

Returns:
bytes or ValueOwnerPair
class skein.kv.ValueOwnerPair

A (value, owner) pair in the key-value store.

Parameters:
value : bytes

The value.

owner : str or None

The owner container_id, or None for no owner.

class skein.kv.count(start=None, end=None, prefix=None)

A request to count keys in the key-value store.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

prefix : str, optional

If provided, will count the number keys matching this prefix.

class skein.kv.count(start=None, end=None, prefix=None)

A request to count keys in the key-value store.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

prefix : str, optional

If provided, will count the number keys matching this prefix.

class skein.kv.list_keys(start=None, end=None, prefix=None)

A request to get a list of keys in the key-value store.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

prefix : str, optional

If provided, will return all keys matching this prefix.

class skein.kv.exists(key)

A request to check if a key exists in the key-value store.

Parameters:
key : str

The key to check the presence of.

class skein.kv.missing(key)

A request to check if a key is not in the key-value store.

This is the inverse of exists.

Parameters:
key : str

The key to check the absence of.

class skein.kv.get(key, default=None, return_owner=False)

A request to get the value associated with a single key.

Parameters:
key : str

The key to get.

default : bytes or None, optional

Default value to return if the key is not present.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.get_prefix(prefix, return_owner=False)

A request to get all key-value pairs whose keys start with prefix.

Parameters:
prefix : str

The key prefix.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.get_range(start=None, end=None, return_owner=False)

A request to get a range of keys.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.pop(key, default=None, return_owner=False)

A request to remove a single key and return its corresponding value.

Parameters:
key : str

The key to pop.

default : bytes or None, optional

Default value to return if the key is not present.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.pop_prefix(prefix, return_owner=False)

A request to remove all key-value pairs whose keys start with prefix, and return their corresponding values.

Parameters:
prefix : str

The key prefix.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.pop_range(start=None, end=None, return_owner=False)

A request to remove a range of keys and return their corresponding values.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.discard(key)

A request to discard a single key.

Returns true if the key was present, false otherwise.

Parameters:
key : str

The key to discard.

class skein.kv.discard_prefix(prefix, return_keys=False)

A request to discard all key-value pairs whose keys start with prefix.

Returns either the number of keys discarded or a list of those keys, depending on the value of return_keys.

Parameters:
prefix : str

The key prefix.

return_keys : bool, optional

If True, the discarded keys will be returned instead of their count. Default is False.

class skein.kv.discard_range(start=None, end=None, return_keys=False)

A request to discard a range of keys.

Returns either the number of keys discarded or a list of those keys, depending on the value of return_keys.

Parameters:
start : str, optional

The lower bound of the key range, inclusive. If not provided no lower bound will be used.

end : str, optional

The upper bound of the key range, exclusive. If not provided, no upper bound will be used.

return_keys : bool, optional

If True, the discarded keys will be returned instead of their count. Default is False.

class skein.kv.put(key, value=no_change, owner=no_change)

A request to assign a value and/or owner for a single key.

Parameters:
key : str

The key to put.

value : bytes, optional

The value to put. Default is to leave value unchanged; an error will be raised if the key doesn’t exist.

owner : str or None, optional

The container id to claim ownership. Provide None to set to no owner. Default is to leave value unchanged.

class skein.kv.swap(key, value=no_change, owner=no_change, return_owner=False)

A request to assign a new value and/or owner for a single key, and return the previous value.

Parameters:
key : str

The key to put.

value : bytes, optional

The value to put. Default is to leave value unchanged; an error will be raised if the key doesn’t exist.

owner : str or None, optional

The container id to claim ownership. Provide None to set to no owner. Default is to leave value unchanged.

return_owner : bool, optional

If True, the owner will also be returned along with the value. Default is False.

class skein.kv.TransactionResult

A result from a key-value store transaction.

Parameters:
succeeded : bool

Whether the transaction conditions evaluated to True.

results : sequence

A sequence of results from applying all operations in the transaction on_success or on_failure parameters, depending on whether the conditions evaluated to True or False.

class skein.kv.value(key)

Represents the value for a key, for use in transaction conditions.

Parameters:
key : str

The key to lookup

class skein.kv.owner(key)

Represents the owner for a key, for use in transaction conditions.

Parameters:
key : str

The key to lookup

class skein.kv.comparison(key, field, operator, rhs)

A comparison of the value or owner for a specified key.

Parameters:
key : str

The corresponding key.

field : {‘value’, ‘owner’}

The field to compare on.

operator : {‘==’, ‘!=’, ‘>’, ‘>=’, ‘<’, ‘<=’}

The comparison operator to use.

rhs : bytes, str or None

The right-hand-side of the condition expression. Must be a bytes if field='value', or str or None if field='owner'.

skein.kv.is_condition(obj)

Return if x is a valid skein key-value store condition

skein.kv.is_operation(obj)

Return if obj is a valid skein key-value store operation

class skein.kv.EventType

Event types to listen on.

Attributes:
ALL : EventType

All events.

PUT : EventType

Only PUT events.

DELETE : EventType

Only DELETE events.

class skein.kv.Event

An event in the key-value store.

Parameters:
key : str

The key affected.

result : ValueOwnerPair or None

The value and owner for the key. None if a 'DELETE' event.

event_type : EventType

The type of event.

event_filter : EventFilter

The event filter that generated the event.

class skein.kv.EventFilter(key=None, prefix=None, start=None, end=None, event_type=None)

An event filter.

Specifies a subset of events to watch for. May specify one of key, prefix, or start/end. If no parameters are provided, selects all events.

Parameters:
key : str, optional

If present, only events from this key will be selected.

prefix : str, optional

If present, only events with this key prefix will be selected.

start : str, optional

If present, specifies the lower bound of the key range, inclusive.

end : str, optional

If present, specifies the upper bound of the key range, exclusive.

event_type : EventType, optional.

The type of event. Default is 'ALL'

class skein.kv.EventQueue(kv)

A queue of events on the key-value store.

Besides the normal Queue interface, also supports iteration.

>>> for event in app.kv.events(prefix='bar'):
...     print(event)

If an event falls into multiple selected filters, it will be placed in the event queue once for each filter. For example, prefix='bar' and key='bart' would both recieve events on key='bart'. If a queue was subscribed to both events, changes to this key would be placed in the queue twice, once for each filter.

All events are unsubscribed when this object is collected. Can also be used as a contextmanager to unsubscribe-all on __exit__, or explicitly call unsubscribe_all.

get(block=True, timeout=None)

Remove and return an item from the queue.

If optional args ‘block’ is true and ‘timeout’ is None (the default), block if necessary until an item is available. If ‘timeout’ is a non-negative number, it blocks at most ‘timeout’ seconds and raises the Empty exception if no item was available within that time. Otherwise (‘block’ is false), return an item if one is immediately available, else raise the Empty exception (‘timeout’ is ignored in that case).

put(item, block=True, timeout=None)

Put an item into the queue.

If optional args ‘block’ is true and ‘timeout’ is None (the default), block if necessary until a free slot is available. If ‘timeout’ is a non-negative number, it blocks at most ‘timeout’ seconds and raises the Full exception if no free slot was available within that time. Otherwise (‘block’ is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (‘timeout’ is ignored in that case).

subscribe(event_filter=None, key=None, prefix=None, start=None, end=None, event_type=None)

Subscribe to an event filter.

May provide either an explicit event filter, or provide arguments to create a new one and add it to the queue. In either case, the event filter is returned.

If no arguments are provided, subscribes to all events.

Parameters:
event_filter : EventFilter

An explicit EventFilter. If provided, no other keyword arguments may be provided.

key : str, optional

If present, only events from this key will be selected.

prefix : str, optional

If present, only events with this key prefix will be selected.

start : str, optional

If present, specifies the lower bound of the key range, inclusive.

end : str, optional

If present, specifies the upper bound of the key range, exclusive.

event_type : EventType, optional.

The type of event. Default is 'ALL'.

Returns:
EventFilter
unsubscribe(event_filter=None, key=None, prefix=None, start=None, end=None, event_type=None)

Unsubscribe from an event filter.

May provide either an explicit event filter, or provide arguments to create a new one and add it to the queue.

If no arguments are provided, unsubscribes from a filter of all events.

A ValueError is raised if the specified filter isn’t currently subscribed to.

Parameters:
event_filter : EventFilter

An explicit EventFilter. If provided, no other keyword arguments may be provided.

key : str, optional

If present, only events from this key will be selected.

prefix : str, optional

If present, only events with this key prefix will be selected.

start : str, optional

If present, specifies the lower bound of the key range, inclusive.

end : str, optional

If present, specifies the upper bound of the key range, exclusive.

event_type : EventType, optional.

The type of event. Default is 'ALL'.

Returns:
EventFilter
unsubscribe_all()

Unsubscribe from all event filters

Application Specification

class skein.ApplicationSpec(services=required, name='skein', queue='default', tags=None, file_systems=None, max_attempts=1)

A complete description of an application.

Parameters:
services : dict

A mapping of service-name to services. At least one service is required.

name : string, optional

The name of the application, defaults to ‘skein’.

queue : string, optional

The queue to submit to. Defaults to the default queue.

tags : set, optional

A set of strings to use as tags for this application.

file_systems : list, optional

A list of Hadoop file systems to acquire delegation tokens for. A token is always acuired for the defaultFS.

max_attempts : int, optional

The maximum number of submission attempts before marking the application as failed. Note that this only considers failures of the application master during startup. Default is 1.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_file(path, format='infer')

Create an instance from a json or yaml file.

Parameters:
path : str

The path to the file to load.

format : {‘infer’, ‘json’, ‘yaml’}, optional

The file format. By default the format is inferred from the file extension.

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_file(path, format='infer', skip_nulls=True)

Write object to a file.

Parameters:
path : str

The path to the file to load.

format : {‘infer’, ‘json’, ‘yaml’}, optional

The file format. By default the format is inferred from the file extension.

skip_nulls : bool, optional

By default null values are skipped in the output. Set to True to output all fields.

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.Service(commands=required, resources=required, instances=1, max_restarts=0, files=None, env=None, depends=None)

Description of a Skein service.

Parameters:
commands : list

Shell commands to startup the service. Commands are run in the order provided, with subsequent commands only run if the prior commands succeeded. At least one command must be provided

resources : Resources

Describes the resources needed to run the service.

instances : int, optional

The number of instances to create on startup. Default is 1.

max_restarts : int, optional

The maximum number of restarts to allow for this service. Containers are only restarted on failure, and the cap is set for all containers in the service, not per container. Set to -1 to allow infinite restarts. Default is 0.

files : dict, optional

Describes any files needed to run the service. A mapping of destination relative paths to File or str objects describing the sources for these paths. If a str, the file type is inferred from the extension.

env : dict, optional

A mapping of environment variables needed to run the service.

depends : set, optional

A set of service names that this service depends on. The service will only be started after all its dependencies have been started.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.FileType

Enum of possible file types to distribute with the application.

Attributes:
FILE : FileType

Regular file

ARCHIVE : FileType

A .zip, .tar.gz, or .tgz file to be automatically unarchived in the containers.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.FileVisibility

Enum of possible file visibilities.

Determines how the file can be shared between containers.

Attributes:
APPLICATION : FileVisibility

Shared only among containers of the same application on the node.

PUBLIC : FileVisibility

Shared by all users on the node.

PRIVATE : FileVisibility

Shared among all applications of the same user on the node.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.File(source=required, type='infer', visibility=FileVisibility.APPLICATION, size=0, timestamp=0)

A file/archive to distribute with the service.

Parameters:
source : str

The path to the file/archive. If no scheme is specified, path is assumed to be on the local filesystem (file:// scheme).

type : FileType or str, optional

The type of file to distribute. Archive’s are automatically extracted by yarn into a directory with the same name as their destination. By default the type is inferred from the file extension.

visibility : FileVisibility or str, optional

The resource visibility, default is FileVisibility.APPLICATION

size : int, optional

The resource size in bytes. If not provided will be determined by the file system.

timestamp : int, optional

The time the resource was last modified. If not provided will be determined by the file system.

classmethod from_dict(obj, **kwargs)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

class skein.Resources(memory=required, vcores=required)

Resource requests per container.

Parameters:
memory : int

The amount of memory to request, in MB. Requests smaller than the minimum allocation will receive the minimum allocation (usually 1024). Requests larger than the maximum allocation will error on application submission.

vcores : int

The number of virtual cores to request. Depending on your system configuration one virtual core may map to a single actual core, or a fraction of a core. Requests larger than the maximum allocation will error on application submission.

classmethod from_dict(obj)

Create an instance from a dict.

Keys in the dict should match parameter names

classmethod from_json(b)

Create an instance from a json string.

Keys in the json object should match parameter names

classmethod from_protobuf(msg)

Create an instance from a protobuf message.

classmethod from_yaml(b)

Create an instance from a yaml string.

to_dict(skip_nulls=True)

Convert object to a dict

to_json(skip_nulls=True)

Convert object to a json string

to_protobuf()

Convert object to a protobuf message

to_yaml(skip_nulls=True)

Convert object to a yaml string

Application Responses

class skein.model.ApplicationState

Enum of application states.

Attributes:
NEW : ApplicationState

Application was just created.

NEW_SAVING : ApplicationState

Application is being saved.

SUBMITTED : ApplicationState

Application has been submitted.

ACCEPTED : ApplicationState

Application has been accepted by the scheduler.

RUNNING : ApplicationState

Application is currently running.

FINISHED : ApplicationState

Application finished successfully.

FAILED : ApplicationState

Application failed.

KILLED : ApplicationState

Application was terminated by a user or admin.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.model.FinalStatus

Enum of application final statuses.

Attributes:
SUCCEEDED : FinalStatus

Application finished successfully.

KILLED : FinalStatus

Application was terminated by a user or admin.

FAILED : FinalStatus

Application failed.

UNDEFINED : FinalStatus

Application has not yet finished.

classmethod values()

The constants of this enum type, in the order they are declared.

class skein.model.ApplicationReport(id, name, user, queue, tags, host, port, tracking_url, state, final_status, progress, usage, diagnostics, start_time, finish_time)

Report of application status.

Parameters:
id : str

The application ID.

name : str

The application name.

user : str

The user that started the application.

queue : str

The application queue.

tags : set of strings

The application tags.

host : str

The host the application master is running on.

port : int

The rpc port for the application master

tracking_url : str

The application tracking url.

state : ApplicationState

The application state.

final_status : FinalStatus

The application final status.

progress : float

The progress of the application, from 0.0 to 1.0.

usage : ResourceUsageReport

Report on application resource usage.

diagnostics : str

The diagnostic message in the case of failures.

start_time : datetime

The application start time.

finish_time : datetime

The application finish time.

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

runtime

The total runtime of the container.

to_protobuf()

Convert object to a protobuf message

class skein.model.ResourceUsageReport(memory_seconds, vcore_seconds, num_used_containers, needed_resources, reserved_resources, used_resources)

Resource usage report.

Parameters:
memory_seconds : int

The total amount of memory (in MBs) the application has allocated times the number of seconds the application has been running.

vcore_seconds : int

The total number of vcores that the application has allocated times the number of seconds the application has been running.

num_used_containers : int

Current number of containers in use.

needed_resources : Resources

The needed resources.

reserved_resources : Resources

The reserved resources.

used_resources : Resources

The used resources.

classmethod from_protobuf(obj)

Create an instance from a protobuf message.

to_protobuf()

Convert object to a protobuf message

Exceptions

exception skein.SkeinError

Bases: Exception

Base class for Skein specific exceptions

exception skein.ConnectionError

Bases: skein.exceptions.SkeinError, ConnectionError

Failed to connect to the daemon or application master

exception skein.DaemonNotRunningError

Bases: skein.exceptions.ConnectionError

The daemon process is not currently running

exception skein.ApplicationNotRunningError

Bases: skein.exceptions.ConnectionError

The application master is not currently running

exception skein.DaemonError

Bases: skein.exceptions.SkeinError

Internal exceptions from the daemon

exception skein.ApplicationError

Bases: skein.exceptions.SkeinError

Internal exceptions from the application master