Link Scraper

Overview

Link Scraper works by parsing the HTML content of a web page to extract all the links. It returns the links in a structured format, including the link URL, title, and more.

Live Test Link Scraper API →

Endpoint

One host, one path per API. The block below shows this call in four languages; every one of them is the same HTTP request. Making requests covers the timeouts, retries and parameter rules that apply to all of them. The SDKs wrap the same call in a typed client.

1
2
3
4
5
6
7
8

curl -X POST https://api.apiverve.com/v1/linkscraper \
  -H "x-api-key: your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

const res = await fetch('https://api.apiverve.com/v1/linkscraper', {
  method: 'POST',
  headers: {
    'x-api-key': 'your_api_key_here',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}),
});

if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);

const { data } = await res.json();
console.log(data);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import requests

res = requests.post(
    "https://api.apiverve.com/v1/linkscraper",
    headers={"x-api-key": "your_api_key_here"},
    json={
    "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
    "maxlinks": 20,
    "includequery": False
},
    timeout=15,
)
res.raise_for_status()

print(res.json()["data"])

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

package main

import (
	"fmt"
	"io"
	"net/http"
	"strings"
)

func main() {
	body := strings.NewReader(`{
  "url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
  "maxlinks": 20,
  "includequery": false
}`)
	req, _ := http.NewRequest("POST", "https://api.apiverve.com/v1/linkscraper", body)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "your_api_key_here")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, _ := io.ReadAll(res.Body)
	fmt.Println(string(out))
}

Replace your_api_key_here with the key from your dashboard. When the inputs arrive as a list rather than one at a time, batch requests run up to 200 of them through this same API in a single call.

Authentication

Send your key in the x-api-key header. That is the only auth step — there is no token exchange and no per-endpoint scope to configure. Authentication covers creating, rotating and revoking keys.

Parameters

Sent as JSON in the request body. Premium parameters are accepted on every plan but only take effect on plans that include them.

Response

Every API returns the same three top-level keys, so one response handler covers your whole integration: status, error and data. Only data changes shape. Response format covers the envelope, the other output formats and how premium fields are withheld.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

{
  "status": "ok",
  "error": null,
  "data": {
    "url": "http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html",
    "linkCount": 16,
    "externalLinkCount": 13,
    "internalLinkCount": 3,
    "links": [
      {
        "text": "Documentation",
        "href": "http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html/index.html",
        "external": false
      },
      {
        "text": "Amazon EC2 Instance Types Guide",
        "href": "https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-types.html",
        "external": true
      },
      {
        "text": "Amazon EC2 Auto Scaling",
        "href": "https://docs.aws.amazon.com/autoscaling/",
        "external": true
      }
    ],
    "uniqueDomains": [
      "docs.aws.amazon.com",
      "aws.amazon.com"
    ],
    "maxLinksReached": false
  }
}

Response fields

Paths are relative to data. Premium fields are absent rather than zeroed on plans that do not include them, so check for presence instead of comparing to 0.

Errors

Read the HTTP status first, then error for the specific reason. The body names the parameter that has to change. Error handling covers the full status list and which of them are worth retrying.

Use cases

Site Architecture Audits

Crawlers inspect internal navigation paths and anchor text across catalog pages to detect orphan URLs and track link equity distribution.

Dead Link Discovery

Run published articles through the scraper to catch external links, resolve relative URLs, and forward destination targets to status checkers.

Competitor Footprint Mapping

Growth analysts feed rival blog posts into the endpoint to inventory citation targets and detect outbound partner referrals.

Web Archival Pipelines

When archiving resource directories, extract all outbound destinations and clean anchor text before caching document references.

Other ways to use Link Scraper

Set up Link Scraper on APIVerve, or reach the same source a different way. Your APIVerve account and credits work on all of them — one key, one balance.

Related

More in Data Scraping: