utils

usas_validator.utils.parse_usas_token_group(usas_tag_group_text, strict=False)

Given a the string that represents the USAS tags whereby each USAS tag is separated by whitespace it is converted into a structured format.

This whitespace separation of USAS tags is the format that is produced by the original C version of the USAS tagger when it outputs USAS tags for a given token or meaningful word unit like a Multi Word Expression (MWE).

The whitespace separation can be one or more spaces, i.e. ` ` or ` `

A USAS tag can be also be PUNCT which represents punctuation. It can also be represented as Df or Df with an affix like +++ or mf etc.

Complex examples of usas_tag_group_text: L1 E3- O4.2- X5.2+ A6.2- A1.7- A7- W3 L2 F1 S1.2.4- Z2 Z2/S2mf Z3 O4.3 G1.2 G1.2/S2mf

Parameters:

• usas_tag_group_text (str) – The string that represents the USAS tags produced by the USAS tagger for one token.

• strict (bool, default: False) – If True, the function will raise an error if the USAS tags within the given text cannot be parsed as a USAS tag (see ValueError below). Default False.

Returns:

Structured format of the USAS tags that can be parsed from the given text. Any text that cannot be parsed as a USAS tag will be ignored and therefore can result in returning an empty list.

Return type:

list[USASTagGroup]

Raises:

ValueError – If strict is True and if the USAS tags within the given text cannot be parsed as a USAS tag, whereby each USAS tag after whitespace and / split should match the following regex: [A-Z](d+)((.d+)+)?, Df, or PUNCT.

Examples

>>> from usas_validator.utils import parse_usas_token_group
>>> usas_token_groups = parse_usas_token_group("Z2/S2mf Z3")
>>> for usas_token_group in usas_token_groups:
...     print(usas_token_group)
tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False), USASTag(tag='S2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=True, male=True, antecedents=False, neuter=False, idiom=False)]
tags=[USASTag(tag='Z3', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]

When using strict=True:

>>> from usas_validator.utils import parse_usas_token_group
>>> parse_usas_token_group("Invalid", strict=True)
Traceback (most recent call last):
    ...
ValueError: Cannot find the tag for this USAS tag text: Invalid

When using strict=False (default) you can ignore invalid USAS tags within the text you are parsing, in the example below Z1 and Z2 are parsed successfully while NONE is ignored:

>>> from usas_validator.utils import parse_usas_token_group
>>> parse_usas_token_group("Z1/NONE Z2", strict=False)
[USASTagGroup(tags=[USASTag(tag='Z1', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]), USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]

usas_validator.utils.load_usas_mapper(usas_tag_descriptions_file, tags_to_filter_out)

Returns a dictionary of USAS tags and their descriptions.

Parameters:

• usas_tag_descriptions_file (Path | None) – The path to the YAML file that contains the USAS tags and their descriptions. If None then the function will use the USAS tags and description file that is located within the package at usas_csv_auto_labeling/data/usas/usas_mapper.yaml.

• tags_to_filter_out (set[str] | None) – A set of USAS tags to filter out.

Returns:

A dictionary of USAS tags and their descriptions.

Return type:

dict[str, str]

Raises:

• FileNotFoundError – If the usas_tag_descriptions_file is not found.

• ValueError – If the usas_tag_descriptions_file is not a file.

Examples

>>> from usas_validator.utils import load_usas_mapper
>>> usas_tag_descriptions = load_usas_mapper(None, None)
>>> usas_tag_descriptions["X1"]
'title: General description: General terms relating to psychological actions, states and processes'

usas_validator.utils.keep_valid_usas_tags(usas_tag_string, valid_usas_tags)

Filter a USAS tag string to only include tags present in a given set of valid tags.

Parses the input string into USASTagGroup objects (in non-strict mode, so malformed tags are skipped rather than raising errors), then removes any individual USASTag whose base tag is not in valid_usas_tags. Tag groups that become empty after filtering are dropped entirely.

Parameters:

• usas_tag_string (str) – A space-separated string of USAS tag groups, e.g. "Z2/S2mf E3-".

• valid_usas_tags (set[str]) – A set of acceptable base USAS tag strings (e.g. {"Z2", "E3"}). Typically obtained from load_usas_mapper().

Return type:

list[USASTagGroup]

Returns:

A list of USASTagGroup objects containing only tags whose base tag appears in valid_usas_tags. Groups with no remaining valid tags are excluded.

Examples

>>> from usas_validator.utils import keep_valid_usas_tags
>>> valid = {"Z2", "E3"}
>>> keep_valid_usas_tags("Z2/S2mf E3-", valid)
[USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]), USASTagGroup(tags=[USASTag(tag='E3', number_positive_markers=0, number_negative_markers=1, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]

>>> keep_valid_usas_tags("Z2/S2mf E3-", {"Z2"})
[USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]

usas_validator.utils.mwe_token_indexes_from_slices(mwe_index_slices)

Expand a list of Multi Word Expression (MWE) index slices into the individual token indexes they cover.

Each tuple is a (start, end) slice, with end exclusive, in the same way as the built-in range callable, e.g. (2, 5) covers indexes 2, 3, and 4. A single token MWE is therefore represented as (i, i + 1), which expands to just i.

Parameters:

mwe_index_slices (list[tuple[int, int]]) – A list of (start, end) tuples, one per MWE, where end is exclusive.

Return type:

frozenset[int]

Returns:

A frozenset of every token index covered by any of the given slices.

Examples

>>> from usas_validator.utils import mwe_token_indexes_from_slices
>>> sorted(mwe_token_indexes_from_slices([(2, 5)]))
[2, 3, 4]

A single token MWE, represented as (i, i + 1):

>>> sorted(mwe_token_indexes_from_slices([(0, 1)]))
[0]

Multiple, possibly overlapping, MWE slices are merged into one frozenset this can occur with MWEs that are discontinuous:

>>> sorted(mwe_token_indexes_from_slices([(0, 2), (1, 3), (5, 6)]))
[0, 1, 2, 5]

usas_validator.utils.mwe_token_labels_from_indexes(mwe_indexes, number_tokens)

Represents each MWE as a unique label, ordered by the starting position of its first token, the first MWE is labelled 1 and the second 2 and so on. Each token’s labels are stored at that token’s index in the returned list.

Parameters:

• mwe_indexes (list[frozenset[int]]) – A list of frozensets of token indexes for each MWE. Every frozenset must be non-empty and every index must satisfy 0 <= index < number_tokens.

• number_tokens (int) – The number of tokens in the sentence.

Return type:

list[set[int]]

Returns:

A list, of length number_tokens, of sets that contain unique labels, one unique label per MWE. A MWE is represented by the tokens that are associated with each label. If a token does not belong to a MWE it is represented as an empty set.

Raises:

ValueError – If any frozenset in mwe_indexes is empty, or if any token index within mwe_indexes is not in the range 0 <= index < number_tokens.

Examples

>>> from usas_validator.utils import mwe_token_labels_from_indexes
>>> mwe_token_labels_from_indexes([frozenset({0, 1, 3}), frozenset({2, 3})], 4)
[{1}, {1}, {2}, {1, 2}]

When a token in the text does not belong to a MWE it is represented as an empty set:

>>> from usas_validator.utils import mwe_token_labels_from_indexes
>>> mwe_token_labels_from_indexes([frozenset({2, 3})], 5)
[set(), set(), {1}, {1}, set()]