TextPreprocessConfig

Configuration for preprocessing text: sentence splitting and length/character filters.

Used by :class:~gradiend.data.text.prediction.creator.TextPredictionDataCreator and :func:preprocess_texts. When preprocess is None (e.g. in the data creator), no preprocessing is applied and texts are used as-is.

Attributes:

Name	Type	Description
split_to_sentences	Union[bool, int]	If True or 1, split input texts into individual sentences (via regex on .!? or, when a spaCy model is provided, via spaCy sentencizer). If an integer greater than 1, join non-overlapping sentence windows of that size, e.g. 2 yields (s1 s2), (s3 s4), .... If False, each input string is treated as one segment. Default False.
min_chars	Optional[int]	Drop segments (sentences or whole texts) with strictly fewer than this many characters. None means no minimum. Default None.
max_chars	Optional[int]	Drop segments with strictly more than this many characters. None means no maximum. Default None.
exclude_chars	Optional[str]	Drop segments that contain any character in this string (e.g. "\x00" to drop nulls). None means no character exclusion. Default None.
custom_filter	Optional[Callable[[str], bool]]	Optional callable (str) -> bool. Only segments for which it returns True are kept. None means no custom filter. Default None.

custom_filter class-attribute instance-attribute

custom_filter = field(default=None, repr=False)

exclude_chars class-attribute instance-attribute

exclude_chars = None

max_chars class-attribute instance-attribute

max_chars = None

min_chars class-attribute instance-attribute

min_chars = None

split_to_sentences class-attribute instance-attribute

split_to_sentences = False

__post_init__

__post_init__()

Source code in gradiend/data/text/preprocess.py

def __post_init__(self) -> None:
    value = self.split_to_sentences
    if isinstance(value, bool):
        return
    if not isinstance(value, int):
        raise TypeError(
            "TextPreprocessConfig.split_to_sentences must be bool or int, "
            f"got {type(value).__name__}"
        )
    if value < 1:
        raise ValueError(
            "TextPreprocessConfig.split_to_sentences integer values must be >= 1, "
            f"got {value}"
        )