BiVelioOneBot: what it reads and how to stop it
It reads a few public pages of a company's website, only when someone asks for it for their company's website or for a client who has given them their consent, and estimates which processes are worth discovering with their team. It follows robots.txt and text and data mining reservations, and always identifies itself.
Who BiVelioOneBot is
It is the web reader of BiVelio One, BiVelio Inc's workspace. It is used by the «Discover your processes from your website» check on bivelio.com and by One's Understanding phase.
Only on request
It does not crawl the Internet on its own: every read starts because a person gives a website and declares that it belongs to their company or to a client who has given them their consent.
One site only
It reads the home page and a few pages of the same site (with or without www). It does not follow links to other domains or read social networks.
Rules, not copies
It turns each page into text, applies explainable rules and discards the HTML. The result is an estimate of processes, never a copy of your website.
What it reads and what it does not do
What it reads
- /robots.txt first and, before analysing anything, /.well-known/tdmrep.json (the text and data mining reservation); then a sitemap: the one robots.txt lists or, if it lists none, /sitemap.xml.
- Up to 8 public HTML pages (home page included), one per topic: contact, services, bookings, jobs, legal notice, FAQ…
- HTML only: it does not download images, PDFs, stylesheets or scripts, and it does not run JavaScript.
- Capped: 2 MiB per page, 8 MiB and 20 seconds in total, at most 3 redirects and always within the same site.
What it never does
- It does not submit forms, log in, create accounts or add anything to any cart.
- It does not keep copies of pages. The bivelio.com check keeps nothing from your website. In One, the company that asked for it keeps the result of the analysis for 90 days (the pages read, the processes and short snippets of at most 240 characters, with emails and phone numbers removed), and a fingerprint (SHA-256) of each page so that what was read can be checked without keeping a copy, after which it is all deleted automatically; never the HTML.
- It sends no cookies, no Authorization header and no Referer.
- It does not work around blocks: on a 403, 429 or 503 (what blocks and almost all challenge pages answer with) it stops, without retrying.
How to exclude your website
Any one of these four ways is enough. BiVelioOneBot checks them before analysing anything, and you do not need to tell us.
robots.txt
It follows robots.txt (RFC 9309) under the name BiVelioOneBot, case-insensitive. To keep it out of your whole website:
User-agent: BiVelioOneBot
Disallow: /Header or meta tag (TDMRep)
Reserve text and data mining with the W3C TDMRep protocol: the HTTP header or the meta tag on your pages.
tdm-reservation: 1
<meta name="tdm-reservation" content="1">/.well-known/tdmrep.json
Or one file for the whole site (or only for the paths you choose), also from TDMRep:
[
{ "location": "/", "tdm-reservation": 1 }
]In your legal notice or terms
If your legal notice or terms of use say that you reserve text and data mining (for example, «Text and data mining of this website is prohibited»), BiVelioOneBot reads it as a reservation and does not analyse the website.
Text and data mining of this website is prohibited.- If there is no group for BiVelioOneBot, it follows the «User-agent: *» group, and the most specific rule wins.
- A noindex on the home page (meta robots or X-Robots-Tag header) stops the analysis; with nofollow it does not follow its links.
- If your robots.txt answers with a 5xx error, it reads nothing; if tdmrep.json answers 429 or 503, it does not either.
- If you would rather it never tried, write to support@bivelio.com from an address on your domain and we will add it to our exclusion list.
Why it respects these signals
The text and data mining exception in Art. 4 of Directive (EU) 2019/790 (in Spain, Art. 67 of Royal Decree-law 24/2021) does not apply when the rightholder has expressly reserved it in an appropriate manner; for content published online, by machine-readable means. BiVelioOneBot treats the W3C TDMRep protocol (header, meta tag and /.well-known/tdmrep.json) as a reservation and also follows robots.txt.
It only reads at the request of someone who declares that the website belongs to their company or to a client who has given them their consent, and the bivelio.com check keeps nothing of what it reads. For any data protection matter: privacy@bivelio.com.
If personal data appear on your website (for example, a name in one of the snippets): BiVelio Inc processes them (in One, on behalf of the client company that asked for the analysis) to estimate the processes of the company that owns the website, on the basis of legitimate interest (Art. 6(1)(f) GDPR; the assessment is available on request at privacy@bivelio.com). They come from the public website itself. The bivelio.com check keeps nothing; in One, at most 90 days, in snippets of 240 characters with no emails or phone numbers. You can request access or erasure, or object, at privacy@bivelio.com, and lodge a complaint with the Spanish Data Protection Agency (aepd.es).
Did it read your website?
If BiVelioOneBot visited your website and you want to know why, want it not to come back, or want whatever remains in One deleted, write to support@bivelio.com (for data protection, privacy@bivelio.com). We reply within 24 working hours.