To find files downloaded in crawler4j

The script tag is never processed. If you want to download. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Use crawler4j to download js files Ask Question. Asked 8 years, 10 months ago. Active 8 years, 9 months ago. Viewed 1k times. Improve this question. Alireza Noori. Alireza Noori Alireza Noori Add a comment. Active Oldest Votes. Improve this answer. When JulienS posted the answer, I was using the exact same method to extract.

However, I thought that maybe modifying the source would help. I think it is better don't touch the original implementation but allow user to implement preferred strategy.

Skip to content. Star 4. New issue. Jump to bottom. Copy link. I don't understand your solution. Your fix downloads only the headers for binary file types. Just don't forget that the actual fetching of headers takes some time as the overhead of networking is used, and I suggest limiting by using the shouldVisit like in BasicExample or by disabling download of binary files altogether or both for best results Avi.

Sign up for free to join this conversation on GitHub. Already have an account? As can be seen in the above code, there are two main functions that should be overridden:. You should also implement a controller class which specifies the seeds of the crawl, the folder in which intermediate crawl data should be stored and the number of concurrent threads:.

The controller class has a mandatory parameter of type CrawlConfig. Instances of this class can be used for configuring crawler4j. The following sections describe some details of configurations. By default there is no limit on the depth of crawling. But you can limit the depth of crawling. For example, assume that you have a seed page "A", which links to "B", which links to "C", which links to "D".

So, we have the following link structure:. Since, "A" is a seed page, it will have a depth of 0. You can set a limit on the depth of pages that crawler4j crawls. For example, if you set this limit to 2, it won't crawl page "D". To set the maximum depth you can use:. Although by default there is no limit on the number of pages to crawl, you can set a limit on this:. By default crawling binary content i.