Sipki crawlers and data tools
Sipki, LLC publishes a small set of data tools that customers run in the cloud. When one of them visits your site, it sends one of the user-agent strings below. The customer decides which sites, companies or files are processed; Sipki, LLC does not receive or store the results.
1. Our user agents
| Token for robots.txt | Full user-agent string | What it does | robots.txt | Rate |
|---|---|---|---|---|
SipkiDomainPassport | SipkiDomainPassport/1.0 (+https://apify.com/sipki/domain-passport) | Builds a technical profile of a domain a customer names: public DNS records (including mail authentication records), one TLS handshake on port 443, the site's robots.txt and at most one home page. | Follows robots.txt. If robots.txt returns a server error or cannot be reached, the home page is not fetched. | 1 page per domain per run; no link following. |
SipkiSEOAudit | SipkiSEOAudit/1.0 (+https://apify.com/sipki/seo-audit-http) | Audits the website a customer names for technical SEO issues: crawls pages of that site and checks outgoing links with HEAD requests. | Follows robots.txt for every page and redirect step and honours Crawl-delay (up to 10 seconds). If robots.txt returns a server error or cannot be reached, the site is not crawled. | 1 request per second per site by default (at most 2); at most 2,000 pages. On HTTP 429 or 503 it pauses as the server asks and stops after three such answers in a row. |
SipkiATSJobs | SipkiATSJobs/1.0 (+https://apify.com/sipki/ats-jobs-api) | Collects public job postings for companies a customer names, through the public job-board APIs of Greenhouse, Lever and Ashby. To find which board a company uses, it may open the company's careers page once. | Follows the company site's robots.txt before opening a careers page; if robots.txt returns a server error or cannot be reached, the page is not opened. | One careers page per company; requests to a job-board API are retried only after the waiting time the server asks for. |
SipkiDocsToText | SipkiDocsToText/1.0 (+https://apify.com/sipki/docs-to-text-ocr) | Converts documents (PDF, DOCX, images) to text. It downloads only the exact file links a customer supplies, once each, and does not crawl or follow links. | Does not read robots.txt, because it is not a crawler: it fetches a single file a person asked for, like a download. To stop it, block its user agent on your server or ask us to exclude your domain (section 3). | One request per supplied file (plus redirects). On HTTP 401, 403 or 429 the file is reported as failed: no retry and no workaround. |
All four tools: no headless browser, no proxies or rotating IP addresses to get around blocks, no CAPTCHA solving, no login to any site, no access to anything behind a password. They read only what any visitor can see. Because they run on shared cloud infrastructure, their IP addresses change; please identify them by user agent, not by IP address.
2. How to block a tool with robots.txt
SipkiDomainPassport, SipkiSEOAudit and SipkiATSJobs follow the group in your robots.txt that names their token, and otherwise the User-agent: * group. To block one of them from your whole site:
User-agent: SipkiSEOAudit
Disallow: /To block all three crawlers:
User-agent: SipkiDomainPassport
User-agent: SipkiSEOAudit
User-agent: SipkiATSJobs
Disallow: /SipkiDocsToText does not read robots.txt; use a server rule or the e-mail opt-out below for it.
Changes take effect from the next run that reads your robots.txt; a running job may keep a copy read at its start. You can also reject requests on your server by matching the user-agent string, for example with an HTTP 403 response; our tools do not try to get around such responses (no proxies, no changed user agent).
3. Opt out by e-mail
If you prefer, or if you want a domain excluded regardless of robots.txt, write to support@sipkigroup.com with the subject “Bot opt-out” and the domain names. Please send it from an address at that domain, or tell us how we can confirm that you control it. We reply within 3 business days and add confirmed domains to an exclusion list that all four tools check before they contact a site.
To report a problem (for example load on your server), include the time, the user-agent string and, if you can, the requested URLs. We investigate every report.
4. Personal data
The tools are not designed to collect personal data, but public pages and documents can contain it. For results produced for a customer, the customer is the controller and Sipki, LLC acts only as its processor. If you want to exercise your rights, see Data requests.