Implement TF–IDF from Scratch (Python + NumPy/SciPy)

You are given a list of documents (strings). Your task is to build a TF–IDF vectorizer from scratch with the following specified behavior.

Requirements

1. Tokenization and Preprocessing:
   • Convert all text to lowercase.
   • Strip punctuation from tokens.
   • Implement a memory-efficient tokenizer (e.g., streaming or generator-based) without using external Natural Language Processing (NLP) libraries.
2. Vocabulary and Filtering:
   • Support optional min_df and max_df document-frequency filtering.
   • These parameters can be interpreted as either:
     • An integer: representing an absolute number of documents.
     • A float in the range (0, 1]: representing a fraction of documents.
   • The vocabulary (which forms the columns of the output matrix) must be ordered lexicographically.
3. TF–IDF Computation:
   • Term Frequency (TF) should be calculated as the raw count of a term within a document.
   • Use the smoothed Inverse Document Frequency (IDF) formula: idf = log((1 + N) / (1 + df)) + 1, where N is the total number of documents and df is the document frequency of the term.
   • The final output must be a SciPy CSR (Compressed Sparse Row) sparse matrix with dimensions (n_documents, vocab_size).
4. Transform Behavior:
   • Ignore any out-of-vocabulary (OOV) tokens encountered during the transform phase.
   • Support optional L2 normalization per row (document) in the output matrix.
5. API Surface:
   • Implement a class with the following methods: fit, transform, fit_transform, and inverse_transform(doc_index, k=None).
   • The inverse_transform method should return the top-k terms for a given document, ordered by their TF–IDF score in descending order. If k is None, return all terms for that document, sorted by score.
6. Complexity Analysis:
   • Analyze and discuss the time and space complexity for both the fit and transform methods. This analysis should be in terms of the number of documents, total tokens, and vocabulary size.
7. Unit Test:
   • Provide a unit test to verify the correctness of your implementation using a small 3-document toy corpus that includes repeated terms.

Constraints

• Only Python standard library, NumPy, and SciPy are allowed. No other external NLP libraries.
• The implementation must gracefully handle empty documents.
• Ensure deterministic lexicographic column order for reproducibility.