When I opened Search Console, the "Why pages aren't indexed" section had a row reading "Page with redirect". With a number next to it, it looks like something that has to be fixed.
On my own website, my first instinct was to dig out every redirect and remove it. But when I walked through the URLs on the list, it included a language-routing redirect that I had placed on purpose, and I could judge that one as needing no action.
Here I will write down the criteria I used to sort redirects into "leave it", "collapse it" and "fix it", plus the script that did the sorting for me. I have also added the instruction I give an agent when I ask it to investigate.
First, what the row actually means
The row describes a state in which the URL is a redirect source, so it is not a candidate for indexing, and the destination is the one that counts. It does not say the redirect itself is wrong.
So the thing to look at is not the source but the destination, and the path that leads there. This is the table I sorted by.
| Kind of redirect | Verdict | Why |
|---|---|---|
| http to https normalisation | No action | The source URL remaining on the list is by design |
| www / non-www normalisation | No action | It exists to pull everything onto one host |
| Trailing-slash normalisation | No action | It folds two spellings of the same page into one |
| Language-based routing | No action (but confirm it is 302/307) | The destination differs per visitor, so a temporary redirect is safer |
| Two or more hops in a row | Collapse into one | Every hop adds delay and one more thing that can break |
| Loop | Fix | It never arrives anywhere |
| Destination is 404 or 5xx | Fix | It throws away whatever the source had earned |
| Destination is noindex | Confirm intent | If deliberate it is correct; if accidental, a page that should be indexed is missing |
What matters is not the count on the screen but being able to say, URL by URL, which row of this table it belongs to.
A script that walks the list one hop at a time
Give it the URLs exported from the report, and it walks each redirect without auto-following, then returns a verdict from the table above. It needs only the standard library.
Two points matter. The first is turning off urllib's auto-follow so that the 3xx comes back as it is; with auto-follow, every intermediate stop disappears. The second is walking each URL twice, once as a plain request and once with Accept-Language: en. The reason comes further down.
#!/usr/bin/env python3
"""redirect_triage.py — triage Search Console's "Page with redirect" list.
Usage: python3 redirect_triage.py urls.txt (one URL per line, exported from the report)
Standard library only. Auto-follow is disabled; each hop is walked by hand."""
import sys, re, urllib.request, urllib.error, urllib.parse
class NoFollow(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *a, **k): # do not follow: surface the 3xx itself as an exception
return None
OPENER = urllib.request.build_opener(NoFollow)
def fetch(url, lang=None):
"""Return (status, location, headers, first bytes of body). Network failure -> status 0."""
req = urllib.request.Request(url, headers={"User-Agent": "redirect-triage/1.0"})
if lang:
req.add_header("Accept-Language", lang)
try:
with OPENER.open(req, timeout=10) as r:
return r.status, None, r.headers, r.read(4096).decode("utf-8", "ignore")
except urllib.error.HTTPError as e:
return e.code, e.headers.get("Location"), e.headers, ""
except Exception:
return 0, None, {}, ""
def chain(url, lang=None, limit=6):
hops, seen = [], set()
while len(hops) < limit:
if url in seen:
return hops, "loop"
seen.add(url)
st, loc, hd, body = fetch(url, lang)
hops.append((url, st))
if st in (301, 302, 303, 307, 308) and loc:
url = urllib.parse.urljoin(url, loc)
continue
noindex = "noindex" in (hd.get("X-Robots-Tag", "") if hd else "").lower() \
or re.search(r'<meta[^>]+name=["\']robots["\'][^>]+noindex', body, re.I)
return hops, ("noindex" if noindex and st == 200 else st)
return hops, "too-long"
def classify(src, hops, end):
if end == "loop": return "NG", "redirect loop"
if end == "too-long": return "NG", "chain is too long"
if isinstance(end, int) and (end >= 400 or end == 0):
return "NG", f"destination returns {end}"
if end == "noindex": return "CHECK", "destination is noindex (confirm it is intended)"
if len(hops) >= 3: return "FIX", f"{len(hops)-1} hops (collapse into one)"
if len(hops) == 1: return "OK", "no redirect"
a, b = urllib.parse.urlsplit(src), urllib.parse.urlsplit(hops[-1][0])
if a.scheme != b.scheme and (a.netloc, a.path) == (b.netloc, b.path):
return "OK", "http to https normalisation"
if a.netloc.replace("www.", "") == b.netloc.replace("www.", "") and a.netloc != b.netloc:
return "OK", "www / non-www normalisation"
if a.path.rstrip("/") == b.path.rstrip("/") and a.path != b.path:
return "OK", "trailing-slash normalisation"
return "CHECK", f"redirects to another URL: {b.path}"
def main(path):
for src in [l.strip() for l in open(path) if l.strip()]:
hops, end = chain(src) # plain request, close to what a crawler sends
hops_en, end_en = chain(src, lang="en") # like an English-language browser
level, note = classify(src, hops, end)
if level == "OK" and (hops[-1][0] != hops_en[-1][0]):
first = hops_en[0][1]
level = "OK" if first in (302, 303, 307) else "FIX"
note = "language-based routing" + (" (temporary redirect, expected)" if level == "OK"
else f" ({first} gets remembered by browsers; use 302/307)")
elif len(hops) == 1 and len(hops_en) > 1:
first = hops_en[0][1]
level = "OK" if first in (302, 303, 307) else "FIX"
note = "language-based routing" + (" (plain request is not redirected, expected)" if level == "OK"
else f" ({first} gets remembered by browsers; use 302/307)")
print(f"{level:5} {src}\n -> {note}")
if __name__ == "__main__":
main(sys.argv[1])The leading word on each output line maps to the verdicts: OK is no action, CHECK is confirm intent, FIX is collapse, and NG is fix.