Why is apnoti.com crawling my website?

smart.apnoti.com is a search engine using a web-indexing robot. Thereby smart.apnoti.com’s web crawler gathers information form the Internet to build a comprehensive and searchable index for search activities using the smart.apnoti.com search engine. These information is revealed and crawled because other websites include links directing to these information and documents.

When performing the crawling procedure , the smart.apnoti.com crawler takes robots.txt principles into account to make sure we do not crawl and index content from those websites whose content you do not want included in smart.apnoti.com search. If a page is prohibited to be crawled by robots.txt principles smart.apnoti.com does not read or use the contents of that page.

go to top

How does smart.apnoti.com index websites?

smart.apnoti.com Robot/v1.34 is the smart.apnoti.com web crawler that automatically crawls the Web to find and add information to the smart.apnoti.com search index. The smart.apnoti.com Robot/v1.34 is a program that scans websites and indexes their content, such as text, documents, images, and links for search services.

While smart.apnoti.com Robot/v1.34 crawls billions of web pages, not every web page is indexed. For a website to be indexed, it must meet specific standards for content, design, and technical implementation. For example, if your website’s link structure doesn't have links to each webpage on your website, smart.apnoti.com Robot/v1.34 might not find all the web pages on your website.

go to top

How do I prevent my site from being crawled?

smart.apnoti.com complies with the Robot Exclusion Standard. In particular, smart.apnoti.com adheres to the 1996 Robots Exclusion Standard (RES).

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable.

smart.apnoti,com obeys the first entry in the robots.txt file with a User-agent containing "smart.apnoti.com Robot/v1.34 (http://smart.apnoti.com/bot)"

• If there is no such record, it will obey the first entry with a User-agent of "*".
• If it is not able to retrieve a robots.txt file, it will assume there are no restrictions for smart.apnoti.com. It will keep trying to retrieve the file, and will obey it if becomes available.

Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed. Smart.apnoti.com does read the home page at each site and uses it internally, but if it is disallowed, it is neither indexed nor followed. If a page has robots.txt standards disallowing it to be crawled, smart.apnoti.com will not read or use the contents of that page.

go to top

How do I stop the search engine bot from indexing certain pages on my site?

You have a certain number of options to prevent smart.apnoti.com from indexing your pages that you don’t want to appear in smart.apnoti.com’s search results.

• If you need to keep confidential content on your server, save it in a password-protected directory. smart.apnoti.com Robot/v1.34 and other spiders won't be able to access these documents and contents. This is the easiest and most effective way to prevent smart.apnoti.com Robot/v1.34 from crawling and indexing content on your website.
• Use a robots.txt to control access to files and directories on your server. The robots.txt file is like an electronic No Trespassing sign. It tells smart.apnoti.com Robot/v1.34 and other crawlers which files and directories on your server are prohibited to be crawled.

go to top