Introduction
hi all
This is a small function for handling POST and GET queries natively in
AHK so external utilities to handle it are not needed any more. It can
be used where no additional utilities, such as cURL or wget are wanted.
The latest version handles following features:
- ports in URL
- Username:Password@domain style URLs (see this answer to avoid common problems)
- SSL aka https
- HeaderInformations / Dumping / Storing
- DownloadSizes / CurrentSize (for download progressbars)
- Flags for InternetConnection handling (auto follow feature etc)
- Referrer for Queries
- Accepttypes for Queries
- Proxy handling
- timeout handling
- custom UserAgent
Usage
Usage is pretty simple:
You need to define a Variable which will receive the returned databuffer.
When only Strings are expected as returnvalues, autohotkey needs to
learn the new size, so a VarSetCapacity(buffer,-1) is needed. This has
been modified to support binary downloads aswell.
httpQUERY(buffer:="",URL) will
result the length of the standard HTML which is usually retrieved. The
html itself is stored in the variable buffer. The query method is GET
as the length of Postparams is zero.
httpQUERY(buffer:="",URL,POSTDATA)
will work as the GET above just with POSTDATA being transmitted. The
query method is POST as the length of Postparams is non zero.
The URL now supports following scheme:
http://username:password@server.tld:80/path/to/target.ext?queryops#fragment
since httpQuery has been updated to use InternetCrackURL from
winINet, all essential parts will be recognized. so there is no need to
set up any additional parameters. Attention:
When u need to authetificate in the Website the username / password
attempt will not work. u have to submit those parameters via POST or
GET method.
Additional Parameters:
To see a dump of the received httpHeaders, there is buildIn support for
a global Variable named httpQueryOps. It may consist of one or more
Verbs. For now "showHeader", "storeHeader", and "updateSize" verbs are
supported. If You use storeHeader the complete Header will be saved in
a variable named HttpQueryHeader which will be made global at runtime.
The verb updateSize will make two variables globally Available:
httpQueryFullSize and httpQueryCurrentSize. An usage example to show a
download status indicator is included
Following Variables are evaluated globally:
httpAgent: can hold an individual UserAgent Code. The default is AutoHotkeyScript
httpProxy: the name of a proxy server to use. default = 0
httpProxyByPass: list of domains which will not be used with the proxy. default = 0
httpQueryReferer: an URL which
is recognized by QueriedServer, as the location where the Request was
generated mind though its "Referrer" the name of the variable is
httpQueryReferer [sic]
httpQueryAcceptType: this variable lets u specify your accepted stream formats for the results of the query
httpQueryDwFlags: if in need
for any special flags for the current connection this is the variable
to set (example V shows an useCase for this feat)
I think these are pretty self explaining.
Examples
Example I: Showing the raw HTML result of a search for httpQuery in AutoHotkey's forum in an edit control
Code (Expand): | ; exmpl.searchAHKforum.httpQuery.ahk ; Searches the forum for a given Phrase: in this case httpQuery #noenv html := "" URL := "http://www.autohotkey.com/forum/search.php?mode=results" POSTData := "search_keywords=httpQuery&search_terms=all&search_forum=-1&" . "search_time=0&search_fields=all&show_results=topics&return_chars=500" . "&sort_by=0&sort_dir=DESC"
length := httpQuery(html,URL,POSTdata) varSetCapacity(html,-1)
Gui,Add,Edit,w600 +Wrap r25,% html Gui,Show Return
GuiClose: GuiEscape: ExitApp #include httpQuery.ahk |
Remarks: To make a function
work with POST parameters, it is essential to know which parameters are
actually sent. These can be obtained by studying the HTML source of the
target URL or by using analyzing tools such as the HttpFox Addon for Firefox. You might also want to use the following example to learn the nature of the form in a HTML document
Example II: Showing a dump of a HTML form to help making valid POST parameters
Code (Expand): | ; exmpl.formdump.httpquery.ahk ; Form Dumper v0.1b (w) 9th July 2008 by derRaphael #NoEnv InputBox ,URL
; OutputVariable ,Enter
URL to analyze
; Title of box ,Please enter the complete URL starting with http:// to be analysed ; Descriptive text ,,,,,,,,http://www.autohotkey.com/forum/search.php ; default value
html := "" htmlLength := httpQuery(html,URL) VarSetCapacity(html,-1)
; The Complete Form Node from given URL's HTML RegExMatch(html,"i)<form.+?</form>",formNode) ; Just the formtag for further analyzing RegExMatch(formNode,"i)<form[^>]+?>",formTag) ; The name of the form RegExMatch(formTag,"i)NAME=""?(?P<Name>.+?)""?>",form) ; The method used to process Data RegExMatch(formTag,"i)METHOD=""?(?P<Method>.+?)""?\s",form) ; The complete address used to send data to RegExMatch(formTag,"i)ACTION=""?(?P<Action>.+?)""?\s",form) ; just the url RegExMatch(formAction,"i)(?P<URL>[^\?]+)",formA_) ; any existing GET parameters RegExMatch(formAction,"i)\?(?P<GET>.*)",formA_) ; Fix & to & as delimiter StringReplace,formA_GET,formA_GET,&`;,&,All
startPosI := startPosS := startPosT := 0 inpCount := selCount := txtCount := 0
Loop, { If (startPosI:=RegExMatch(formNode,"i)<input[^>]+>",inputTag,startPosI+1)) { inpCount++ formInput%InpCount% := inputTag } else { noMoreInput := 1 } If (startPosS:=RegExMatch(formNode,"i)<select.*?</select>",selectNode,startPosS+1)) { selCount++ formSelect%selCount% := selectNode } else { noMoreSelectOptions := 1 } If (startPosT:=RegExMatch(formNode,"i)<textarea.*?</textarea>",textareaNode,startPosT+1)) { txtCount++ formTextarea%txtCount% := textareaNode } else { noMoreTextArea := 1 } if (NoMoreInput) && (NoMoreTextarea) && (NoMoreSelectOptions) { break } } Loop,% inpCount Inputs .= formInput%A_Index% "`n" StringReplace,inputs,inputs,<,%A_Tab%<,All Loop,% selCount Selects .= formSelect%A_Index% "`n" StringReplace,Selects,Selects,</option>,</option>`n,All StringReplace,Selects,Selects,<option,%A_Tab%<option,All Selects := RegExReplace(selects,"i)<select[^>]+>","$0`n") Loop,% txtCount txtAreas .= formTextArea%A_Index% "`n" StringReplace,txtAreas,txtAreas,</textarea>,</textarea>`n,All StringReplace,txtAreas,txtAreas,<textarea,%A_Tab%<textarea,All
; dump results to Gui Gui,Add,Tab2, w800 h20,Analyzed Content|found Form|raw HTML Gui,Tab,1 Gui,Add,Edit,w800 h600 yp+20 xp , % "" . "formTag:`n" formTag "`n`n" . "formMethod:`t" formMethod "`n" . "formAction:`t" formAction "`n" . "formActionURL:`t" formA_URL "`n" . "formActionGET:`t" formA_GET "`n`n" . "Total Inputs:`t" inpCount "`n" . inputs . "`nTotal Selects:`t" selCount "`n" . selects "`n" . "`nTotal TextAreas:`t" txtCount "`n" . txtAreas "`n" Gui,Tab,2 Gui,Add,Edit,w800 h600 yp xp , % formNode Gui,Tab,3 Gui,Add,Edit,w800 h600 yp xp , % html Gui,Show,,Statistics for Form: "%formName%" (%URL%) return
GuiClose: GuiEscape: ExitApp #include httpQuery.ahk
|
Remarks: Using this function provides informations of which elements are used in a form which will be submitted. It neither catches modded JavaScript content nor more than one form. It has been tested with google.com, autohotkey.com's searchform and autohotkey.net's pastebin
Example III: Downloading binary data and saving it to a local file.
Code (Expand): | ; exmpl.downloadBinary.httpQuery.ahk ; This example downloads the latest AHK environment and stores ; the received binary data to a file. #noenv data := "" URL := "http://www.autohotkey.net/programs/AutoHotkey104706.zip" httpQueryOps := "updateSize" SetTimer,showSize,10 length := httpQuery(data,URL) Tooltip if (write_bin(data,"ahk.exe",length)!=1) MsgBox "There was an Error!" else MsgBox AHK Source downloaded and saved as "ahk.zip"! Return
showSize: Tooltip,% HttpQueryCurrentSize "/" HttpQueryFullSize return
GuiClose: GuiEscape: ExitApp
write_bin(byref bin,filename,size){ h := DllCall("CreateFile","str",filename,"Uint",0x40000000 ,"Uint",0,"UInt",0,"UInt",4,"Uint",0,"UInt",0) IfEqual h,-1, SetEnv, ErrorLevel, -1 IfNotEqual ErrorLevel,0,ExitApp ; couldn't create the file r := DllCall("SetFilePointerEx","Uint",h,"Int64",0,"UInt *",p,"Int",0) IfEqual r,0, SetEnv, ErrorLevel, -3 IfNotEqual ErrorLevel,0, { t = %ErrorLevel% ; save ErrorLevel to be returned DllCall("CloseHandle", "Uint", h) ErrorLevel = %t% ; return seek error } result := DllCall("WriteFile","UInt",h,"Str",bin,"UInt" ,size,"UInt *",Written,"UInt",0) h := DllCall("CloseHandle", "Uint", h) return, 1 }
#include httpQuery.ahk |
Remarks: This example
downloads the current Autohotkey Source and saves it to local disk as
ahk.zip in the same path as the script. While it does so, it shows the
current progress of the download
Example IV: Uploading an image to Imageshack using the official (free) API
Code (Expand): | ; exmpl.imageshack.httpQuery.ahk ; This example uploads an image and constructs a multipart/form-data Type ; for fileuploading and returns the XML which is returned to show the stored Imagepath FileSelectFile,image FileGetSize,size,%image% SplitPath,image,OFN FileRead,img,%image% VarSetCapacity(placeholder,size,32) boundary := makeProperBoundary() post:="--" boundary "`ncontent-disposition: form-data; name=""MAX_FILE_SIZE""`n`n" . "1048576`n--" boundary "`ncontent-disposition: form-data; name=""xml""`n`nyes`n--" . boundary "`ncontent-disposition: form-data; name=""fileupload""; filename=""" . ofn """`nContent-type: " MimeType(img) "`nContent-Transfer-Encoding: binary`n`n" . placeholder "`n--" boundary "--" headers:="Content-type: multipart/form-data, boundary=" boundary "`nContent-Length: " strlen(post) DllCall("RtlMoveMemory","uInt",(offset:=&post+strlen(post)-strlen(Boundary)-size-5) ,"uInt",&img,"uInt",size) size := httpQuery(result:="","http://www.imageshack.us/index.php",post,headers) VarSetCapacity(result,-1) Gui,Add,Edit,w800 h600, % result Gui,Show return
GuiClose: GuiEscape: ExitApp
makeProperBoundary(){ Loop,26 n .= chr(64+a_index) n .= "0123456789" Loop,% StrLen(A_Now) { Random,rnd,1,% StrLen(n) Random,UL,0,1 b .= RegExReplace(SubStr(n,rnd,1),".$","$" (round(UL)? "U":"L") "0") } Return b }
MimeType(ByRef Binary) { MimeTypes:="424d image/bmp|4749463 image/gif|ffd8ffe image/jpeg|89504e4 image/png|4657530" . " application/x-shockwave-flash|49492a0 image/tiff" @:="0123456789abcdef" Loop,8 hex .= substr(@,(*(a:=&Binary-1+a_index)>>4)+1,1) substr(@,((*a)&15)+1,1) Loop,Parse,MimeTypes,| if ((substr(hex,1,strlen(n:=RegExReplace(A_Loopfield,"\s.*"))))=n) Mime := RegExReplace(A_LoopField,".*?\s") Return (Mime!="") ? Mime : "application/octet-stream" }
#include httpQuery.ahk |
Remarks: Uploading binary
data where an encodingtype "multipart/form-data" was needed, didnt work
with classic POST usage. this example shows how to build the complete
POST request in a different manner and to include raw binary data with
no encoding such as base64 or ascii85
Example V: Modifying dwFlags for special useCases where a 302 is the returncode such as with dynamic created content urls
Code (Expand): | #NoEnv Gui,add,edit,w800 h400 vPasteBin, this is a testtext! Gui,Add,Button,gPaste wp, Upload this text to http://pastebin.com Gui,Show return
Paste: Gui,Submit,Nohide URL := "http://pastebin.com/pastebin.php" POSTDATA := "parent_pid=&format=text&code2=" uriEncode(PasteBin) "&" . "poster=&paste=Send&expiry=d&email=&" httpQueryOps := "showHeader storeHeader" httpQueryDwFlags := (INTERNET_FLAG_NO_AUTO_REDIRECT:= 0x00200000) length := httpQuery(HTML:="",URL,POSTDATA) VarSetCapacity(HTML,-1) gui,destroy Gui,Add,Edit,w800 h600,% HttpQueryHeader "`n`n" html Gui,Show Return
GuiClose: GuiEscape: ExitApp
uriEncode(str) { ; v 0.3 / (w) 24.06.2008 by derRaphael / zLib-Style release b_Format := A_FormatInteger data := "" SetFormat,Integer,H Loop,Parse,str if ((Asc(A_LoopField)>0x7f) || (Asc(A_LoopField)<0x30) || (asc(A_LoopField)=0x3d)) data .= "%" . ((StrLen(c:=SubStr(ASC(A_LoopField),3))<2) ? "0" . c : c) Else data .= A_LoopField SetFormat,Integer,%b_format% return data }
#include httpQUERY.ahk |
Remarks: Some WebServices
such as PasteBin.com generate Dynamic Location Urls in order to show
processed content. Using httpQuery in a default manner, results in a
404 Not found error when the script tries to access the location to
fast. Specifying the NO_FOLLOW dwFlag solves this problem but needs a
2nd request to grab the fresh received Location Header.
|
|
|