Englishবাংলা
SubscribeWrite
Latest
Big-O notation explained: time and space complexity for interviewsSoftware testing basics: the test pyramid and what to testClean code and code review habits that make you a better developerREST API design best practices: URLs, status codes and errorsJavaScript and TypeScript roadmap: from first script to full stackPython learning roadmap: from basics to real projectsRead documentation and debug like a developerFrontend developer roadmap: HTML, CSS, JavaScript and beyond

Web archives and public sources: finding forgotten pages the right way

Sites change, but the past is archived. Learn how public archives and open sources help you understand a target, and where to draw the line.

••2 min read
An archive shelf with old web pages

Applications grow over time. Features are added, renamed or removed, but old pages, endpoints and documents often remain reachable, and copies of them live in public archives. Looking at these sources is OSINT (open-source intelligence): using only information that is publicly available.

Why history matters

  • Old endpoints may still work. A page removed from the menu may still respond.
  • Old API versions (/v1/) can be less protected than the current one.
  • Parameters and file names used in the past hint at how the app is built.
  • Documentation describes features that the UI does not show.

Public sources to read

Web archives. The Internet Archive's Wayback Machine and similar services keep snapshots of pages over time. You can browse old versions of a site and list archived URLs for a domain. Old JavaScript files in archives can show endpoints that no longer appear in the current front end.

Search engines. Careful use of search operators shows what a search engine has indexed for a domain: documentation pages, uploaded documents, staging or test pages that were linked by mistake.

Public code repositories. Developers and companies publish code. Searching a company's public repositories (and its employees' public projects, only where relevant and public) can show API paths, configuration names and sometimes accidentally published secrets.

Documentation and developer portals. Public API docs, changelogs, release notes, status pages and help centres describe features and sometimes internal terms.

Mobile apps. The public app package contains endpoints and settings used by the mobile client.

How to work with it

  1. Collect URLs and endpoint names from the sources above.
  2. Filter to hosts and paths that are in scope.
  3. Check which still respond, gently and within the program's automation rules.
  4. Compare with the current app and note differences (old features, old API versions).
  5. Test the interesting ones only with your own accounts, in scope.

Handling sensitive finds

Sometimes a public source contains something that should not be public: a private key, a database password, internal documents, personal data.

  • Do not use it to log in or access systems. Using exposed credentials is usually outside the rules and can be illegal.
  • Do not download more than needed to prove it exists, and do not share it.
  • Report it to the program (or the owner) with the location and why it is sensitive, and let them rotate or remove it.
  • Do not post it publicly.

Limits and ethics

  • Old content may now be out of scope or belong to a different company after an acquisition. Check ownership.
  • Archived pages may contain other people's personal data. Do not collect it.
  • OSINT is passive. Once you start sending requests to a host, the program's rules for active testing apply.

Takeaway: archives, search engines, public repositories and documentation reveal how an app evolved. Use them to find forgotten surface that is in scope, and report any exposed secret instead of using it.

Bug Bounty series, Part 2 (recon and tools): 4 of 5. Previous: Burp Suite basics: proxy, repeater and a safe testing workflow.

Keep reading