Jaccard Similarity of Two Strings

Given two input strings, a and b, compute their Jaccard similarity based on their respective token sets.

Tokenization Rules

To derive the token set for each string, apply the following rules:

• Convert the entire string to lowercase.
• Split the string into tokens using any non-alphabetic character as a delimiter (e.g., spaces, punctuation marks).
• Discard any resulting empty tokens.
• Treat each string as a set of unique tokens, meaning duplicate tokens within a single string should be counted only once.

Jaccard Similarity Formula

Let A be the token set derived from string a, and B be the token set derived from string b.

The Jaccard similarity J(A,B) is defined as:

J(A,B) = |A ∩ B| / |A ∪ B|

Where:

• |A ∩ B| is the size of the intersection of sets A and B (number of common unique tokens).
• |A ∪ B| is the size of the union of sets A and B (total number of unique tokens across both sets).

Output

Return the calculated similarity as a floating-point number.

Edge Cases

• If both token sets A and B are empty, the similarity should be defined as 1.0.

Example

• a = "I like coffee, coffee"
• b = "coffee is great"

1. Tokenization of a:

   • Lowercase: "i like coffee, coffee"
   • Split on non-alphabetic: ["i", "like", "coffee", "", "coffee"]
   • Discard empty: ["i", "like", "coffee", "coffee"]
   • Unique set A: {i, like, coffee}

2. Tokenization of b:

   • Lowercase: "coffee is great"
   • Split on non-alphabetic: ["coffee", "is", "great"]
   • Discard empty: ["coffee", "is", "great"]
   • Unique set B: {coffee, is, great}

3. Calculate Jaccard Similarity:

   • Intersection A ∩ B: {coffee}. Size = 1.
   • Union A ∪ B: {i, like, coffee, is, great}. Size = 5.
   • Similarity = 1 / 5 = 0.2

Return 0.2.