javascript - Scraping dynamically generated html inside Android app - Stack Overflow

admin管理员组

文章数量:1340885

I am currently writing an Android app that, among other things, uses text information from websites which I do not own. In addition, some of the pages require authentification.

For some pages I have been able to log in and retrieve the html code using BasicNameValuePairs and an HTTPClient with its associated objects.

Unfortunately, these methods retrieve the webpage source without running any javascript functions that a browser (Android Webview even) would normally run. I need the text that some of these scripts are retrieving.

I've done my research, but everything I've found is guesswork & extremely confusing. I'm okay with ignoring pages that require login for now. Also, I am willing to post any code that may be useful for constructing a solution; It is an independent project.

Any concrete solutions for scraping the html result from javascript calls? An example would be absolutely top-notch.

I am currently writing an Android app that, among other things, uses text information from websites which I do not own. In addition, some of the pages require authentification.

For some pages I have been able to log in and retrieve the html code using BasicNameValuePairs and an HTTPClient with its associated objects.

Unfortunately, these methods retrieve the webpage source without running any javascript functions that a browser (Android Webview even) would normally run. I need the text that some of these scripts are retrieving.

I've done my research, but everything I've found is guesswork & extremely confusing. I'm okay with ignoring pages that require login for now. Also, I am willing to post any code that may be useful for constructing a solution; It is an independent project.

Any concrete solutions for scraping the html result from javascript calls? An example would be absolutely top-notch.

• javascript
• android
• html
• web-scraping

Share Improve this question Follow edited Jun 21, 2012 at 15:45 bhekman asked Jun 19, 2012 at 1:08 bhekmanbhekman 3,2772323 silver badges2424 bronze badges 1

• Note: Many threads I've seen suggest looking into "Rhino", but I haven't found a way to make it work yet. Rhino: mozilla/rhino/tutorial.html – bhekman Commented Jun 19, 2012 at 1:11

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 8

Final Success:

• Rhino. Used this jar file.

Other Things I Tried:

• HttpClient provided by Android
  • Cannot run javascript
• HtmlUnit
  • 4 hours, no success. Also huge, added 12 mb to my apk.
• SL4A
  • Finally piled. Used THIS guide to set-up. Abandoned as overkill for a simple rhino jar.

Things That Might Work:

• Selenium

Further results will be posted. Others results will be added if posted.

Note: many of the options listed above reference each other. I think rhino is included in both sl4a and htmlunit. Also, I think htmlunit contains selenium.

The aforementioned solutions are very slow and restrict you to 1 url (well, not really, but I dare you to scrape 10 urls with Rhino while your user is impatiently waiting for results).

An alternative is to use a cloud scraping solution. You get the benefit of not wasting phone bandwidth on downloading content you won't use.

Try this solution: Bobik Java SDK

It gives you the ability to scrape up to hundreds of sites in a matter of seconds

本文标签： javascriptScraping dynamically generated html inside Android appStack Overflow