mlblocks.discovery

Primitives and Pipelines discovery module.

This module contains functions to load primitive and pipeline annotations, as well as to configure how MLBlocks finds the primitives and pipelines.

Functions:

add_pipelines_path(path)

Add a new path to look for pipelines.

add_primitives_path(path)

Add a new path to look for primitives.

find_pipelines([pattern, filters])

Find pipelines by name and filters.

find_primitives([pattern, filters])

Find primitives by name and filters.

get_pipelines_paths()

Get the list of folders where pipelines will be looked for.

get_primitives_paths()

Get the list of folders where primitives will be looked for.

load_pipeline(name)

Locate and load the pipeline JSON annotation.

load_primitive(name)

Locate and load the primitive JSON annotation.

mlblocks.discovery.add_pipelines_path(path)[source]

Add a new path to look for pipelines.

The new path will be inserted in the first place of the list, so any primitive found in this new folder will take precedence over any other pipeline with the same name that existed in the system before.

Parameters

path (str) – path to add

Raises

ValueError – A ValueError will be raised if the path is not valid.

mlblocks.discovery.add_primitives_path(path)[source]

Add a new path to look for primitives.

The new path will be inserted in the first place of the list, so any primitive found in this new folder will take precedence over any other primitive with the same name that existed in the system before.

Parameters

path (str) – path to add

Raises

ValueError – A ValueError will be raised if the path is not valid.

mlblocks.discovery.find_pipelines(pattern='', filters=None)[source]

Find pipelines by name and filters.

If a patter is given, only the pipelines whose name matches the pattern will be returned.

If filters are given, they should be a dictionary containing key/value filters that will have to be matched within the pipeline annotation for it to be included in the results.

If the given key is not found but it contains dots, split by the dots and consider each part a sublevel in the annotation.

If the key value within the annotation is a list or a dict, check whether any of the given values is contained within it instead of checking for equality.

Parameters
  • pattern (str) – Regular expression to match agains the pipeline names.

  • filters (dict) – Dictionary containing the filters to apply over the matchin pipelines.

Returns

Names of the matching pipelines.

Return type

list

mlblocks.discovery.find_primitives(pattern='', filters=None)[source]

Find primitives by name and filters.

If a patter is given, only the primitives whose name matches the pattern will be returned.

If filters are given, they should be a dictionary containing key/value filters that will have to be matched within the primitive annotation for it to be included in the results.

If the given key is not found but it contains dots, split by the dots and consider each part a sublevel in the annotation.

If the key value within the annotation is a list or a dict, check whether any of the given values is contained within it instead of checking for equality.

Parameters
  • pattern (str) – Regular expression to match agains the primitive names.

  • filters (dict) – Dictionary containing the filters to apply over the matchin primitives.

Returns

Names of the matching primitives.

Return type

list

mlblocks.discovery.get_pipelines_paths()[source]

Get the list of folders where pipelines will be looked for.

This list will include the values of all the entry points named pipelines published under the entry point group mlblocks.

An example of such an entry point would be:

entry_points = {
    'mlblocks': [
        'pipelines=some_module:SOME_VARIABLE'
    ]
}

where the module some_module contains a variable such as:

SOME_VARIABLE = os.path.join(os.path.dirname(__file__), 'jsons')
Returns

The list of folders.

Return type

list

mlblocks.discovery.get_primitives_paths()[source]

Get the list of folders where primitives will be looked for.

This list will include the values of all the entry points named primitives published under the entry point group mlblocks.

Also, for backwards compatibility reasons, the paths from the entry points named jsons_path published under the mlprimitives group will also be included.

An example of such an entry point would be:

entry_points = {
    'mlblocks': [
        'primitives=some_module:SOME_VARIABLE'
    ]
}

where the module some_module contains a variable such as:

SOME_VARIABLE = os.path.join(os.path.dirname(__file__), 'jsons')
Returns

The list of folders.

Return type

list

mlblocks.discovery.load_pipeline(name)[source]

Locate and load the pipeline JSON annotation.

All the pipeline paths will be scanned to find a JSON file with the given name, and as soon as a JSON with the given name is found it is returned.

Parameters

name (str) – Path to a JSON file or name of the JSON to look for withouth the .json extension.

Returns

The content of the JSON annotation file loaded into a dict.

Return type

dict

Raises

ValueError – A ValueError will be raised if the pipeline cannot be found.

mlblocks.discovery.load_primitive(name)[source]

Locate and load the primitive JSON annotation.

All the primitive paths will be scanned to find a JSON file with the given name, and as soon as a JSON with the given name is found it is returned.

Parameters

name (str) – Path to a JSON file or name of the JSON to look for withouth the .json extension.

Returns

The content of the JSON annotation file loaded into a dict.

Return type

dict

Raises

ValueError – A ValueError will be raised if the primitive cannot be found.