utils
- usas_validator.utils.parse_usas_token_group(usas_tag_group_text, strict=False)[source]
Given a the string that represents the USAS tags whereby each USAS tag is separated by whitespace it is converted into a structured format.
This whitespace separation of USAS tags is the format that is produced by the original C version of the USAS tagger when it outputs USAS tags for a given token or meaningful word unit like a Multi Word Expression (MWE).
The whitespace separation can be one or more spaces, i.e. ` ` or ` `
A USAS tag can be also be PUNCT which represents punctuation. It can also be represented as Df or Df with an affix like +++ or mf etc.
Complex examples of usas_tag_group_text: L1 E3- O4.2- X5.2+ A6.2- A1.7- A7- W3 L2 F1 S1.2.4- Z2 Z2/S2mf Z3 O4.3 G1.2 G1.2/S2mf
- Parameters:
- Returns:
Structured format of the USAS tags that can be parsed from the given text. Any text that cannot be parsed as a USAS tag will be ignored and therefore can result in returning an empty list.
- Return type:
- Raises:
ValueError – If strict is True and if the USAS tags within the given text cannot be parsed as a USAS tag, whereby each USAS tag after whitespace and / split should match the following regex: [A-Z](d+)((.d+)+)?, Df, or PUNCT.
Examples
>>> from usas_validator.utils import parse_usas_token_group >>> usas_token_groups = parse_usas_token_group("Z2/S2mf Z3") >>> for usas_token_group in usas_token_groups: ... print(usas_token_group) tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False), USASTag(tag='S2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=True, male=True, antecedents=False, neuter=False, idiom=False)] tags=[USASTag(tag='Z3', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]
When using strict=True:
>>> from usas_validator.utils import parse_usas_token_group >>> parse_usas_token_group("Invalid", strict=True) Traceback (most recent call last): ... ValueError: Cannot find the tag for this USAS tag text: Invalid
When using strict=False (default) you can ignore invalid USAS tags within the text you are parsing, in the example below Z1 and Z2 are parsed successfully while NONE is ignored:
>>> from usas_validator.utils import parse_usas_token_group >>> parse_usas_token_group("Z1/NONE Z2", strict=False) [USASTagGroup(tags=[USASTag(tag='Z1', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]), USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]
- usas_validator.utils.load_usas_mapper(usas_tag_descriptions_file, tags_to_filter_out)[source]
Returns a dictionary of USAS tags and their descriptions.
- Parameters:
usas_tag_descriptions_file (
Path|None) – The path to the YAML file that contains the USAS tags and their descriptions. If None then the function will use the USAS tags and description file that is located within the package at usas_csv_auto_labeling/data/usas/usas_mapper.yaml.tags_to_filter_out (
set[str] |None) – A set of USAS tags to filter out.
- Returns:
A dictionary of USAS tags and their descriptions.
- Return type:
- Raises:
FileNotFoundError – If the usas_tag_descriptions_file is not found.
ValueError – If the usas_tag_descriptions_file is not a file.
Examples
>>> from usas_validator.utils import load_usas_mapper >>> usas_tag_descriptions = load_usas_mapper(None, None) >>> usas_tag_descriptions["X1"] 'title: General description: General terms relating to psychological actions, states and processes'
- usas_validator.utils.keep_valid_usas_tags(usas_tag_string, valid_usas_tags)[source]
Filter a USAS tag string to only include tags present in a given set of valid tags.
Parses the input string into
USASTagGroupobjects (in non-strict mode, so malformed tags are skipped rather than raising errors), then removes any individualUSASTagwhose base tag is not invalid_usas_tags. Tag groups that become empty after filtering are dropped entirely.- Parameters:
usas_tag_string (
str) – A space-separated string of USAS tag groups, e.g."Z2/S2mf E3-".valid_usas_tags (
set[str]) – A set of acceptable base USAS tag strings (e.g.{"Z2", "E3"}). Typically obtained fromload_usas_mapper().
- Return type:
- Returns:
A list of
USASTagGroupobjects containing only tags whose base tag appears invalid_usas_tags. Groups with no remaining valid tags are excluded.
Examples
>>> from usas_validator.utils import keep_valid_usas_tags >>> valid = {"Z2", "E3"} >>> keep_valid_usas_tags("Z2/S2mf E3-", valid) [USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)]), USASTagGroup(tags=[USASTag(tag='E3', number_positive_markers=0, number_negative_markers=1, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]
>>> keep_valid_usas_tags("Z2/S2mf E3-", {"Z2"}) [USASTagGroup(tags=[USASTag(tag='Z2', number_positive_markers=0, number_negative_markers=0, rarity_marker_1=False, rarity_marker_2=False, female=False, male=False, antecedents=False, neuter=False, idiom=False)])]
- usas_validator.utils.mwe_token_indexes_from_slices(mwe_index_slices)[source]
Expand a list of Multi Word Expression (MWE) index slices into the individual token indexes they cover.
Each tuple is a (start, end) slice, with end exclusive, in the same way as the built-in
rangecallable, e.g. (2, 5) covers indexes 2, 3, and 4. A single token MWE is therefore represented as (i, i + 1), which expands to just i.- Parameters:
mwe_index_slices (
list[tuple[int,int]]) – A list of (start, end) tuples, one per MWE, where end is exclusive.- Return type:
- Returns:
A frozenset of every token index covered by any of the given slices.
Examples
>>> from usas_validator.utils import mwe_token_indexes_from_slices >>> sorted(mwe_token_indexes_from_slices([(2, 5)])) [2, 3, 4]
A single token MWE, represented as (i, i + 1):
>>> sorted(mwe_token_indexes_from_slices([(0, 1)])) [0]
Multiple, possibly overlapping, MWE slices are merged into one frozenset this can occur with MWEs that are discontinuous:
>>> sorted(mwe_token_indexes_from_slices([(0, 2), (1, 3), (5, 6)])) [0, 1, 2, 5]
- usas_validator.utils.mwe_token_labels_from_indexes(mwe_indexes, number_tokens)[source]
Represents each MWE as a unique label, ordered by the starting position of its first token, the first MWE is labelled 1 and the second 2 and so on. Each token’s labels are stored at that token’s index in the returned list.
- Parameters:
mwe_indexes (
list[frozenset[int]]) – A list of frozensets of token indexes for each MWE. Every frozenset must be non-empty and every index must satisfy 0 <= index < number_tokens. A singleton frozenset, e.g. {2}, is treated as a single-token MWE (see the single token example below), consistent with the (i, i + 1) slice convention used bymwe_token_indexes_from_slices(); this function does not distinguish genuine multi-token MWEs from single-token ones.number_tokens (
int) – The number of tokens in the sentence.
- Return type:
- Returns:
A list, of length number_tokens, of sets that contain unique labels, one unique label per MWE. A MWE is represented by the tokens that are associated with each label. If a token does not belong to a MWE it is represented as an empty set.
- Raises:
ValueError – If any frozenset in mwe_indexes is empty, or if any token index within mwe_indexes is not in the range 0 <= index < number_tokens.
Examples
>>> from usas_validator.utils import mwe_token_labels_from_indexes >>> mwe_token_labels_from_indexes([frozenset({0, 1, 3}), frozenset({2, 3})], 4) [{1}, {1}, {2}, {1, 2}]
When a token in the text does not belong to a MWE it is represented as an empty set:
>>> from usas_validator.utils import mwe_token_labels_from_indexes >>> mwe_token_labels_from_indexes([frozenset({2, 3})], 5) [set(), set(), {1}, {1}, set()]
A singleton frozenset is labelled the same way as any other MWE, even though it represents a single token rather than a multi word expression:
>>> from usas_validator.utils import mwe_token_labels_from_indexes >>> mwe_token_labels_from_indexes([frozenset({0}), frozenset({1, 2})], 3) [{1}, {2}, {2}]
- usas_validator.utils.mwe_labels_from_pymusas_indexes(pymusas_mwe_index_slices)[source]
Convert the raw, per-token Multi Word Expression (MWE) index slices produced by a PyMUSAS tagger into MWE labels for each token in the sentence.
For each token, PyMUSAS reports a list of (start, end) slices, one per token in the MWE it belongs to (more than one slice means the MWE is discontinuous), following the same convention as
mwe_token_indexes_from_slices(). Single word expressions are reported as a single (i, i + 1) slice and are not MWEs, so they are filtered out here before labelling the remainder withmwe_token_labels_from_indexes().- Parameters:
pymusas_mwe_index_slices (
list[list[tuple[int,int]]]) – One entry per token in the sentence, in token order, each being the list of (start, end) MWE index slices PyMUSAS reported for that token (e.g. from token._.pymusas_mwe_indexes).- Return type:
- Returns:
A list, of the same length as pymusas_mwe_index_slices, of sets that contain unique MWE labels per token, as returned by
mwe_token_labels_from_indexes(). Tokens that are not part of a MWE are represented as an empty set.
Examples
>>> from usas_validator.utils import mwe_labels_from_pymusas_indexes >>> mwe_labels_from_pymusas_indexes([[(0, 2)], [(0, 2)], [(2, 3)]]) [{1}, {1}, set()]
Single token expressions, represented as (i, i + 1), are not MWEs and are therefore not labelled:
>>> from usas_validator.utils import mwe_labels_from_pymusas_indexes >>> mwe_labels_from_pymusas_indexes([[(0, 1)], [(1, 2)]]) [set(), set()]
Discontinuous MWEs report more than one slice per token, but still resolve to a single label:
>>> from usas_validator.utils import mwe_labels_from_pymusas_indexes >>> mwe_labels_from_pymusas_indexes( ... [[(0, 1), (2, 3)], [(1, 2)], [(0, 1), (2, 3)]] ... ) [{1}, set(), {1}]