This article is about our legacy product - Crawler. It is currently implemented as a public actor. We don't recommend using it anymore unless you alrelady have crawlers set up and don't want to migrate. For everybody else we recommend using modern and maintained Web Scraper or build a custom actor using our javascript SDK.
In this tutorial, we will demonstrate step-by-step how to set up a basic crawler with Apify. We recommend you try the steps yourself in a separate browser window. All you need is a basic knowledge of HTML, JavaScript, CSS and ideally jQuery.
What is our goal?
...
First, please sign in to Apify and and go to the Crawlers section, where you will see copies of example crawlers from our front page. Click the Add new button to create a new crawler.
So, let's dive in. First, fill in Custom ID to give a name to your new crawler, for example "Hacker News - my very first crawler". Then add "https://news.ycombinator.com/news" to Start URLs to let the crawler know which web page it should open and click save button. And that's all you need to perform the first dry run of your crawler, because Page function is pre-defined for you..
Note that the crawler only loaded a single web page, because we didn't tell it how to find the next pages - we will address this issue later.
Now let's have a look at the Results tab, which contains the structured data extracted from the page. As you can see, it only contains dummy values because we only used the pre-filled Page function that did not extract any meaningful values from the web page. In the next chapter, we will fix this.
Did this answer your question?
