javascript - downloading a file from a webpage using python script without url , calling onClick function - Stack Overflow

admin管理员组

文章数量:1323335

There is a webpage which have a link "Click to Download" Clicking which a file is downloaded . I can download this file manually by going to webpage and clicking on this link however I need to download this file via a python script .

If i see the source i can see the anchor tag is will run a js function

<a class="download-data-link1" onclick=" document.forms['dataform'].submit()" style="cursor:pointer; vertical-align: middle;">Download in csv</a>

There is a webpage which have a link "Click to Download" Clicking which a file is downloaded . I can download this file manually by going to webpage and clicking on this link however I need to download this file via a python script .

If i see the source i can see the anchor tag is will run a js function

<a class="download-data-link1" onclick=" document.forms['dataform'].submit()" style="cursor:pointer; vertical-align: middle;">Download in csv</a>

But i dont know the url of csv file and i am looking for a way to download it via python .

I know we can download a file if we have url using httplib but couldnt understand how to get a file without url .

Tried few things like in header added 'Content-Disposition': 'attachment;filename="data.csv"'}

but it dosent seems to work . Any ideas ?

• javascript
• python
• html

Share Follow asked Jan 26, 2015 at 9:09 paarth batrapaarth batra 1,41255 gold badges3232 silver badges5454 bronze badges 4

• It has to python? Because links have a tendency (especially nowadays) to be added to the DOM later on, or to respond to certain events, triggered by the user, which makes a headless browser a better option here (IMHO). I'd look into PhantomJS or similar tools – Elias Van Ootegem Commented Jan 26, 2015 at 9:13
• Well even if its not python maybe something which i can invoke from python script . Just saw phantomjs as per your suggestion . Need to study it further – paarth batra Commented Jan 26, 2015 at 9:18
• PhantomJS can be automated through selenium - so it can be python. – alecxe Commented Jan 26, 2015 at 9:19
• @paarthbatra: Selenium + chromedriver would be an option, too (link is to 2 scripts: get-drivers downloads what you need, codeception-start starts/stops selenium), you can then use that setup with a python script quite easily – Elias Van Ootegem Commented Jan 26, 2015 at 9:26

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 3

Thanks all for your answers but I want to add, how i implemented it.

1. First of all you can create a firefox profile. To do that:
2. Close all firefox browsers
3. go to cmd prompt and execute firefox.exe -P
4. create a profile and note down the name of the folder where new profile is created

You can set some options for your profile here, like - automatically download these kind of files from content etc.

Now Download selenium for python and use below code

import os
from selenium import webdriver

download_dir="D:\a"

fp = webdriver.FirefoxProfile(<profile directory here as in step 4>)
fp.set_preference("browser.download.dir", download_dir)
fp.set_preference("browser.download.manager.showWhenStarting",False)
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream");

browser = webdriver.Firefox(firefox_profile=fp)
browser.get("http://pypi.python/pypi/selenium")
# you can use your url here 
browser.find_element_by_partial_link_text("selenium-2").click()
# Use your method to identify class or link text here
browser.close();

Hope this may help others :)

Two basic options can be applied here:

• mimic the logic involved in the onclick() call - in your case, make the dataform form submission using requests, or mechanize

• high-level approach - automate a real browser, headless (PhantomJS) or not, using selenium - find the link and click it:

  from selenium import webdriver
  
  driver = webdriver.PhantomJS()
  driver.get('url here')
  
  driver.find_element_by_class_name('download-data-link1').click()
  

Though, as far as I understand, clicking the link would trigger a "Download" browser dialog to appear - then PhantomJS is not an option since it doesn't support downloads. In case of Chrome or Firefox you would need to tweak browser capabilities to automatically download files without opening the popup, see:

• Access to file download dialog in Firefox
• Firefox + Selenium WebDriver and download a csv file automatically

本文标签：