Turn Mixed Text Into a Clean, Deduplicated Domain Table

Reliable domain extraction from emails and URLs in text is a pipeline, not a single regular expression. Detect likely candidates, classify them, parse each candidate on the appropriate path, isolate the email domain or URL host, normalize it, optionally derive the registrable domain with Public Suffix List-aware tooling, and only then deduplicate.

The safest default output is a structured dataset—not an unexplained list of strings. Keep the original match, source type, parsed host, normalized host, registrable domain, public suffix, status, and any rejection reason.

Related: How to Embed CSV in a Website: Choose a Method.

The reliable extraction workflow in six steps

Use this sequence:

1. Detect candidates: Find likely standalone emails and absolute http:// or https:// URLs.
2. Classify them: Label each candidate by source type and resolve overlapping matches.
3. Parse by type: Apply email-specific checks to emails and a URL parser to URLs.
4. Isolate the host: Extract the domain portion from a controlled email candidate or read the URL parser’s host field.
5. Normalize and classify: Normalize case and internationalized hosts, distinguish DNS domains from IP addresses and local names, and optionally derive the registrable domain.
6. Deduplicate: Remove duplicates using the selected normalized field rather than the raw match.

Standards become useful after a URL candidate has been identified. RFC 3986 defines generic URI syntax, including components such as scheme, authority, user information, host, port, path, query, and fragment. It does not provide a complete method for discovering URL-like substrings in arbitrary prose. Discovery remains an application-level task involving boundaries, punctuation, markup, and false-positive policy.

Emails and URLs therefore need separate handling paths. Do not expect one expression to discover, parse, validate, and classify every possible email address, URL, hostname, and registrable domain.

Compact pseudocode makes the control flow explicit:

records = []
seen = set()

for candidate in scan_candidates(text):
    if overlaps_higher_priority_candidate(candidate):
        continue

    if candidate.type == "email":
        result = parse_simple_email(candidate.text)
    else if candidate.type == "absolute_url":
        result = parse_absolute_url(candidate.text)
    else:
        result = reject("unsupported candidate type")

    if result.has_host:
        result = classify_host(result)

        if result.host_type == "dns_domain":
            result.normalized_host = normalize_host(result.host)

            if registrable_domain_requested:
                result = apply_public_suffix_lookup(result)

    if result.accepted:
        dedupe_key = select_dedupe_key(result)

        if dedupe_key in seen:
            result.status = "duplicate"
        else:
            seen.add(dedupe_key)

    records.append(result)

This is control-flow guidance, not a complete standards-conforming implementation. The record for each candidate should preserve enough context to explain why it was accepted, rejected, deduplicated, or classified separately.

Choose what “domain” means before extracting anything

“Domain” can refer to several values:

• Full host: The complete parsed host, including subdomains.
• Subdomain: Labels to the left of the registrable domain.
• Registrable domain: The public suffix plus the label immediately to its left.
• Public suffix: The portion under which registrations commonly occur, such as com or co.uk.

For news.bbc.co.uk:

• Full host: news.bbc.co.uk
• Subdomain: news
• Registrable domain: bbc.co.uk
• Public suffix: co.uk

Choose the output according to the analysis:

Analytical goal	Value to keep	Example
Compare individual services	Full host	news.bbc.co.uk
Group related hosts	Registrable domain	bbc.co.uk
Classify by suffix	Public suffix	co.uk
Preserve future options	Host and registrable domain	Both values

A registrable domain can be useful as a grouping key, but it is not proof that the grouped records belong to one organization.

Removing www is only prefix cleanup. It does not remove arbitrary subdomains or determine the registrable domain. Selecting the final two labels also fails for multipart suffixes: news.bbc.co.uk would incorrectly become co.uk. Fixed-length and final-two-label approaches are documented as failing on longer top-level domains and varying suffix structures in this technical discussion of regex-based domain extraction.

Use explicit column names such as full_host, registrable_domain, and public_suffix. A column named only domain leaves readers guessing which level it contains.

Discover candidates without asking regex to do everything

Use pragmatic regular expressions—or another scanner—to locate likely emails and absolute HTTP or HTTPS URLs. Treat every discovery match as a candidate, not a validated result.

Candidate discovery must account for boundaries around:

• commas and sentence-ending periods;
• straight and curly quotation marks;
• HTML tags and attributes;
• Markdown link delimiters;
• unmatched closing brackets or parentheses;
• adjacent semicolons and exclamation marks.

Do not blindly remove every punctuation character at the end of a match. A practical scanner can trim obviously unmatched delimiters while retaining the original text and source span for review.

An anchored domain regex can help check an already-isolated hostname. It is not directly suitable for finding hostnames throughout arbitrary prose. Removing anchors from some whole-string validators can also create excessive backtracking and ReDoS exposure; an independent technical discussion of domain validation recommends segmenting or parsing input before applying an anchored check.

A conservative default discovers:

• likely standalone email addresses;
• absolute http:// URLs;
• absolute https:// URLs.

This default misses relative URLs such as ../docs/page.html, bare domains such as example.com, and scheme-less forms such as www.example.com. Add those as separate discovery modes only when the use case requires them, because they provide less context and can create more false positives.

Parse email domains and URL hosts on separate paths

For a controlled, already-isolated email candidate, require one @ separator with nonempty content on both sides. Reject multiple separators, whitespace, and obviously malformed domain content. The email domain is the portion after @, but splitting alone is not complete email validation; simple first-separator approaches can accept multiple separators or an empty domain, as shown by the limitations of this basic email-domain extraction method.

This is an intentionally limited policy for ordinary addresses. It does not promise coverage of every email syntax permitted by applicable email specifications.

For URLs, use a standards-based parser and read its host component. Do not split manually on /, @, or :. Consider:

https://user:pass@news.bbc.co.uk:8443/story?q=1#top

Structured parsing separates:

scheme:   https
userinfo: user:pass
host:     news.bbc.co.uk
port:     8443
path:     /story
query:    q=1
fragment: top

The WHATWG URL Standard defines URL and host parsing behavior, including parsing failures, validation errors, IP addresses, and internationalized domain processing. A pipeline still needs an acceptance policy because a validation error does not always terminate parsing.

A mailto: URI requires scheme-specific handling. Its payload may contain an email address, but that address is not the generic URL host. Route it through the email path.

Input	Handling	Result
Alice@Mail.Example.co.uk	Simple email path	mail.example.co.uk
https://user:pass@news.bbc.co.uk:8443/story?q=1#top	Parse URL host	news.bbc.co.uk
name@@example.com	Reject multiple @ signs	No host
name@	Reject empty domain	No host
../docs/page.html	Relative URL without base	No host

A relative URL yields a host only after resolution against a usable, trusted base URL. Without that context, record it as relative and unresolved instead of inventing a domain.

Derive registrable domains with Public Suffix List-aware tooling

A URL parser identifies the host. It does not necessarily identify the registrable domain, public suffix, or subdomain.

Consider:

support.google.co.jp

Taking the final two labels produces co.jp, which is the suffix rather than the registrable domain. Suffix-aware processing can return:

• Full host: support.google.co.jp
• Subdomain: support
• Registrable domain: google.co.jp
• Public suffix: co.jp

Documented suffix-aware examples likewise distinguish full-host extraction, public-suffix extraction, removal of www, and conversion of support.google.co.jp to google.co.jp for grouping (Tinybird’s technical overview).

Fixed assumptions based on two- or three-character top-level domains fail because top-level domains can be longer and public suffixes can contain multiple labels. Use a maintained, Public Suffix List-aware implementation rather than hardcoding suffixes or selecting a fixed number of labels. Confirm whether the chosen tool uses the suffix rules appropriate to your project. For reproducible processing, record the dependency and suffix-data versions when the implementation exposes them.

Suffix-aware classification answers a structural question only.

Normalize, classify, and deduplicate without losing meaning

Lowercase URL hosts and email-domain portions for comparison. Preserve the original match, and do not automatically lowercase the entire email address: normalization of the local part is broader than the domain-extraction task requires.

Document these policy decisions:

• whether a trailing fully qualified domain dot is removed;
• whether www remains in full_host;
• whether internationalized hosts are stored as Unicode, ASCII Punycode, or both;
• which field supplies the deduplication key;
• whether first-seen order is preserved.

For internationalized hosts, a practical model is to store a stable ASCII representation in normalized_host and a readable value in unicode_host. Do not use display formatting alone as the comparison key.

Classify the parsed host before treating it as a DNS domain. A parser can return an IPv4 address, IPv6 address, localhost, a private single-label name, or an opaque host.

Candidate type	Default policy	Treatment
Absolute HTTP/HTTPS URL	Accept	Parse and classify host
Standalone email	Conditional	Apply controlled email checks
mailto: URI	Conditional	Route payload to email handling
Relative URL	Separate	Resolve only with trusted base
Bare domain	Optional	Use higher-risk discovery mode
IP literal	Separate	Do not derive registrable domain
localhost or private name	Separate	Do not imply public DNS status
Unicode host	Conditional	Store ASCII and Unicode forms
Malformed candidate	Reject	Preserve rejection reason

Keep parsing success, policy validity, Public Suffix List recognition, DNS existence, and network reachability as separate fields or checks.

Deduplicate only after normalization. If source sequence matters, use a set for membership checks while retaining the first accepted occurrence in order. Keep later records with a duplicate status when auditability matters.

Create a publication-ready domain dataset

A practical schema includes:

Column	Purpose
source_row	Input row or document reference
original_match	Exact candidate text
candidate_type	Email, URL, relative URL, IP, or other
full_host	Host before policy reductions
normalized_host	Stable comparison value
registrable_domain	Suffix-aware grouping value
public_suffix	Recognized suffix
unicode_host	Optional Unicode display value
status	Accepted, duplicate, rejected, or separate
rejection_reason	Explanation for non-accepted rows

The following synthetic review data shows the candidate-level fields:

Row	Original match	Candidate type	Status
1	Analyst@Example.com	email	accepted
2	https://user:pass@news.bbc.co.uk:8443/story	URL	accepted
3	https://support.google.co.jp/help	URL	accepted
4	https://01-БЕЗОПАСНОСТЬ.рф/	URL	accepted
5	https://EXAMPLE.COM/about	URL	duplicate
6	../docs/page.html	relative URL	separate
7	http://127.0.0.1:8080/	URL/IP	separate
8	name@@example.com	email	rejected

Store the normalized host and display form separately:

Row	Full host	Normalized host	Unicode host
1	Example.com	example.com
2	news.bbc.co.uk	news.bbc.co.uk
3	support.google.co.jp	support.google.co.jp
4	01-БЕЗОПАСНОСТЬ.рф	xn--01--8cdeyo3chcizco4m.xn--p1ai	01-БЕЗОПАСНОСТЬ.рф
5	EXAMPLE.COM	example.com
6
7	127.0.0.1	127.0.0.1
8

The remaining classification fields make the decisions auditable:

Row	Registrable domain	Public suffix	Rejection reason
1	example.com	com
2	bbc.co.uk	co.uk
3	google.co.jp	co.jp
4	xn--01--8cdeyo3chcizco4m.xn--p1ai	xn--p1ai
5	example.com	com	Duplicate normalized host
6			No trusted base URL
7			Parsed host is an IP address
8			Multiple @ separators

Retaining rejected and separately classified rows prevents ambiguous input from disappearing silently. Review the output before export, then save the cleaned result as CSV or another structured format.

Use synthetic or already-public inputs for demonstrations, and do not publish confidential email addresses, credentials, private logs, or sensitive source text.

Before publishing:

• Define whether you need full hosts, registrable domains, public suffixes, or all three.
• Discover candidates without treating discovery as validation.
• Parse emails and URLs on separate paths.
• Use suffix-aware classification when registrable domains are required.
• Normalize before deduplication.
• Preserve provenance, original matches, statuses, and rejection reasons.
• Publish only a reviewed, nonsensitive table.

Does an extracted domain prove that the website exists or belongs to a particular organization?

No. Extraction and normalization establish only what the input appears to contain under the selected parsing policy. Registration, DNS resolution, reachability, ownership, and safety require separate checks.

Should confidential text be pasted into an online domain extractor?

Not without reviewing and approving how that specific service handles the data. At least one documented extraction tool processes text or retrieved pages on its server, illustrating why processing location matters for confidential material (RAKKOTOOLS documentation). For private logs, internal documents, credentials, or personal data, use an approved processing environment and publish only the cleaned, nonsensitive output.