Counts the frequency of each word in the input text.

This function splits the input text into words (sequences of alphanumeric characters) and returns a dictionary where keys are words (converted to lowercase) and values are their respective counts in the text.

Arguments

• text - The input text to analyze

Returns

• PyResult<HashMap<String, u32>> - A dictionary mapping words to their counts

Examples

from fastpy_rs import ai

text = "Hello hello world! This is a test. Test passed!"
result = ai.token_frequency(text)
# Returns: {'hello': 2, 'world': 1, 'this': 1, 'is': 1, 'a': 1, 'test': 2, 'passed': 1}

Performance

This function is implemented in Rust for high performance, making it significantly faster than equivalent Python implementations, especially for large texts.