featurehub.evaluation package

Submodules

featurehub.evaluation.client module

class featurehub.evaluation.client.EvaluatorClient(problem_id, username, orm, dataset={}, target=None, entities_featurized=None)

Bases: object

check_if_registered(feature, verbose=False)

Check if feature is registered.

Extracts source code, then looks for the identical source code in the feature database.

feature : function verbose : bool

Whether to print output.

evaluate(feature)

Evaluate feature on training dataset and return key performance metrics.

Runs the feature in an isolated environment to extract the feature values. Validates the feature values. Then, builds a model on that one feature and computes key cross-validated metrics. Prints results and returns a dictionary with (metric => value) entries. If the feature is invalid, prints reason and returns empty dictionary.

feature : function

Feature to evaluate

submit(feature, description)

Submit feature to server for evaluation on test data.

If successful, registers feature in feature database and returns key performance metrics.

Runs the feature in an isolated environment to extract the feature values. Validates the feature values. Then, builds a model on that one feature, performs cross validation, and returns key performance metrics.

feature : function

Feature to evaluate

description : str

Feature description

class featurehub.evaluation.client.EvaluatorServer(problem_id, username, orm)

Bases: featurehub.evaluation.client.EvaluatorClient

check_if_registered(code, verbose=False)

Check if feature is registered.

Overwrites client method by expecting code to be passed directly. This is because on the server, we are limited to be unable to do code -> function -> code.

code : str verbose : bool, optional (default=False)

Whether to print output.

evaluate(feature)

Evaluate feature.

Returns a dictionary with (metric => value) entries. If the feature is invalid, re-raises the ValueError.

feature : function

Feature to evaluate

submit(feature, description)

Does nothing.

This class is instantiated at the server, thus we are already registering the feature.

featurehub.evaluation.discourse module

class featurehub.evaluation.discourse.DiscourseFeatureTopic(feature, metrics)

Bases: object

format_code()

format_metrics()

get_params()

post_feature()

class featurehub.evaluation.discourse.DiscourseFeatureTopicTemplate

Bases: object

bullet = ' * '

indent = ' '

render(problem_name, feature_description, feature_code, feature_metrics, user_name)

render_title(title)

featurehub.evaluation.future module

class featurehub.evaluation.future.HubAuth(**kwargs)

Bases: jupyterhub.services.auth.HubAuth

cache

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

Subclasses can declare default classes by overriding the klass attribute

cache_max_age

An int trait.

user_for_token(token, use_cache=True)

Ask the Hub to identify the user for a given token. Args:

token (str): the token use_cache (bool): Specify use_cache=False to skip cached cookie values (default: True)

Returns:

user_model (dict): The user model, if a user is identified, None if authentication fails. The ‘name’ field contains the user’s name.

featurehub.evaluation.response module

class featurehub.evaluation.response.EvaluationResponse(status_code='okay', metrics=None, topic_url='')

Bases: flask.wrappers.Response

Wrapper class for response from evaluation server.

status_code : string, optional (default=EvaluationResponse.STATUS_CODE_OKAY)

Possible values are EvaluationResponse static members.

metrics : list of Metric, optional (default=None)

List of Metric objects, which each encode metric name, metric scoring method, and value.

topic_url : str

>>> EvaluationResponse(status_code=EvaluationResponse.STATUS_CODE_SERVER_ERROR)

STATUS_CODE_BAD_AUTH = 'bad_auth'

STATUS_CODE_BAD_FEATURE = 'bad_feature'

STATUS_CODE_BAD_REQUEST = 'bad_request'

STATUS_CODE_DB_ERROR = 'db_error'

STATUS_CODE_DUPLICATE_FEATURE = 'duplicate_feature'

STATUS_CODE_OKAY = 'okay'

STATUS_CODE_SERVER_ERROR = 'server_error'

classmethod from_string(string)

Instantiate EvaluationResponse from a json-dumped string.

This is useful for recreating the instance on the receiving end of the web connection.

string : str

Json-dumped string encoding the response.

featurehub.evaluation.server module

Module contents