Downloading Files: Difference between revisions
(Replace page with information on the new HTTP support.) |
No edit summary |
||
(12 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
'''QB64-PE''' starting in '''v3.5.0''' includes built-in support for making HTTP requests and receiving their responses. This functionality is implemented in [[_OPENCLIENT]], and the resulting HTTP handle can then be provided the various existing or new stream commands to interact with the response. | |||
QB64-PE | {{Text|'''Before QB64-PE v4.0.0 HTTP functionality was unstable and requires [[$UNSTABLE]]:HTTP to be able to use it.'''|red}} | ||
==Starting the Connection== | {| align="right" style="max-width:25%;" | ||
| __TOC__ | |||
|} | |||
== Starting the Connection == | |||
[[_OPENCLIENT]] is used to make an HTTP request, via using the '''HTTP:''' scheme. The string provided to [[_OPENCLIENT]] should be of the form "'''HTTP:'''url", where '''url''' is the HTTP url you want to make a request too. This can include anything valid in a URL, such as a port, resource path, query string, or fragment. The call to [[_OPENCLIENT]] will then return a handle to the HTTP connection which can be provided to all the rest of the commands related to HTTP requests. If the HTTP request cannot be made (for example, the domain is invalid) then zero is returned instead of a valid handle. | |||
== Metadata About the HTTP Response == | |||
The below commands all give metadata about the HTTP response. They're ordered in terms of usefulness: | The below commands all give metadata about the HTTP response. They're ordered in terms of usefulness: | ||
* [[_STATUSCODE]] can be used to get the HTTP status code given in the HTTP response (Ex. 200, 404, 500, ...) | * [[_STATUSCODE]] can be used to get the HTTP status code given in the HTTP response (Ex. 200, 404, 500, ...) | ||
* [[LOF]] will return the 'Content Length' of the HTTP response if it was provided. It gives -1 if it is not known. | * [[LOF]] will return the 'Content Length' of the HTTP response if it was provided. It gives -1 if it is not known. | ||
Line 20: | Line 18: | ||
* [[_CONNECTIONADDRESS$]] can be used to get the "Effective URL" that was connected too, after taking redirects into account. | * [[_CONNECTIONADDRESS$]] can be used to get the "Effective URL" that was connected too, after taking redirects into account. | ||
* [[_CONNECTED]] can be used to determine whether the HTTP connection to the server is still open. | * [[_CONNECTED]] can be used to determine whether the HTTP connection to the server is still open. | ||
In general for most HTTP responses you should ignore [[_CONNECTED]] and only make use of [[EOF]]. [[EOF]] will continue to return false until you have read all the data from the HTTP response and the connection has been closed, where-as [[_CONNECTED]] will return false even if you still have data to read. | In general for most HTTP responses you should ignore [[_CONNECTED]] and only make use of [[EOF]]. [[EOF]] will continue to return false until you have read all the data from the HTTP response and the connection has been closed, where-as [[_CONNECTED]] will return false even if you still have data to read. | ||
== Reading the HTTP Response Content == | |||
[[GET (HTTP statement)|GET]] is used to read the data from the HTTP response. It should be used with a variable length string, which will then be resized by [[GET (HTTP statement)|GET]] so that it contains all the data current available for the HTTP response. You can then either parse the data as it comes, or collect it all together to parse later. Data is available for reading with [[GET (HTTP statement)|GET]] until [[EOF]] returns true. A [[_LIMIT]] or similar delay should be put in place when looping and using [[GET (HTTP statement)|GET]] so that you don't use up too much CPU time. The actual downloading happens on a separate thread in your program, so how often you call [[GET (HTTP statement)|GET]] has no impact on how fast the download is. | [[GET (HTTP statement)|GET]] is used to read the data from the HTTP response. It should be used with a variable length string, which will then be resized by [[GET (HTTP statement)|GET]] so that it contains all the data current available for the HTTP response. You can then either parse the data as it comes, or collect it all together to parse later. Data is available for reading with [[GET (HTTP statement)|GET]] until [[EOF]] returns true. A [[_LIMIT]] or similar delay should be put in place when looping and using [[GET (HTTP statement)|GET]] so that you don't use up too much CPU time. The actual downloading happens on a separate thread in your program, so how often you call [[GET (HTTP statement)|GET]] has no impact on how fast the download is. | ||
== Closing The HTTP handle == | |||
Like any handle returned from [[_OPENCLIENT]], all HTTP handles need to be closed by using [[CLOSE]] when you are finished with them. | Like any handle returned from [[_OPENCLIENT]], all HTTP handles need to be closed by using [[CLOSE]] when you are finished with them. | ||
{{PageExamples}} | {{PageExamples}} | ||
;Example 1:This function provides basic download functionality. The downloaded data will returned in a string, and the statuscode is provided so that you can check whether the request was successful. | |||
{{CodeStart}} | {{CodeStart}} | ||
' Content of the HTTP response is returned. | ' Content of the HTTP response is returned. | ||
' The statusCode is also assigned. | ' The statusCode is also assigned. | ||
{{Cl|FUNCTION}} Download$(url | {{Cl|FUNCTION}} Download$(url {{Cl|AS}} {{Cl|STRING}}, statusCode {{Cl|AS}} {{Cl|LONG}}) | ||
{{Cl|DIM}} h | {{Cl|DIM}} h {{Cl|AS}} {{Cl|LONG}}, content {{Cl|AS}} {{Cl|STRING}}, s {{Cl|AS}} {{Cl|STRING}} | ||
h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | ||
Line 47: | Line 43: | ||
{{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | {{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | ||
{{Cl|_LIMIT}} 60 | {{Cl|_LIMIT}} 60 | ||
GET #h, , s | {{Cl|GET (HTTP statement)|GET}} #h, , s | ||
content = content + s | content = content + s | ||
{{Cl|WEND}} | {{Cl|WEND}} | ||
Line 57: | Line 53: | ||
{{CodeEnd}} | {{CodeEnd}} | ||
---- | |||
;Example 2:This shows how you could modify the basic usage to provide a progress bar. Note that displaying progress is only possible if the server gives a `Content-length` header back in the response, which will be indicating to you via the result of [[LOF]] on the handle: | |||
{{CodeStart}} | {{CodeStart}} | ||
{{Cl|FUNCTION}} DownloadWithProgress$(url | {{Cl|FUNCTION}} DownloadWithProgress$(url {{Cl|AS}} {{Cl|STRING}}, statusCode {{Cl|AS}} {{Cl|LONG}}) | ||
{{Cl|DIM}} h | {{Cl|DIM}} h {{Cl|AS}} {{Cl|LONG}}, content {{Cl|AS}} {{Cl|STRING}}, s {{Cl|AS}} {{Cl|STRING}}, length {{Cl|AS}} {{Cl|LONG}} | ||
{{Cl|DIM}} progress | {{Cl|DIM}} progress {{Cl|AS}} {{Cl|DOUBLE}} | ||
h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | ||
Line 72: | Line 69: | ||
{{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | {{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | ||
{{Cl|_LIMIT}} 60 | {{Cl|_LIMIT}} 60 | ||
GET #h, , s | {{Cl|GET (HTTP statement)|GET}} #h, , s | ||
content = content + s | content = content + s | ||
Line 96: | Line 93: | ||
{{CodeEnd}} | {{CodeEnd}} | ||
---- | |||
;Example 3:This shows processing multiple downloads at the same time, with progress for each (Again keeping in mind that progress only works if [[LOF]] gives a valid length): | |||
{{CodeStart}} | {{CodeStart}} | ||
{{Cl|SUB}} DownloadMultipleWithProgress() | {{Cl|SUB}} DownloadMultipleWithProgress() | ||
{{Cl|DIM}} handles(1 To 3) | {{Cl|DIM}} handles(1 To 3) {{Cl|AS}} {{Cl|LONG}} | ||
{{Cl|DIM}} content(1 To 3) | {{Cl|DIM}} content(1 To 3) {{Cl|AS}} {{Cl|STRING}} | ||
{{Cl|DIM}} length(1 to 3) | {{Cl|DIM}} length(1 to 3) {{Cl|AS}} {{Cl|LONG}} | ||
{{Cl|DIM}} i | {{Cl|DIM}} i {{Cl|AS}} {{Cl|LONG}}, progress {{Cl|AS}} {{Cl|DOUBLE}} | ||
{{Cl|DIM}} EndOfFile | {{Cl|DIM}} EndOfFile {{Cl|AS}} {{Cl|LONG}}, s {{Cl|AS}} {{Cl|STRING}} | ||
handles(1) = {{Cl|_OPENCLIENT}}("HTTP:https://www.google.com") | handles(1) = {{Cl|_OPENCLIENT}}("HTTP:<nowiki>https://www.google.com</nowiki>") | ||
handles(2) = {{Cl|_OPENCLIENT}}("HTTP:https://www.google.com") | handles(2) = {{Cl|_OPENCLIENT}}("HTTP:<nowiki>https://www.google.com</nowiki>") | ||
handles(3) = {{Cl|_OPENCLIENT}}("HTTP:https://www.google.com") | handles(3) = {{Cl|_OPENCLIENT}}("HTTP:<nowiki>https://www.google.com</nowiki>") | ||
{{Cl|FOR}} i = 1 To {{Cl|UBOUND}}(handles) | {{Cl|FOR}} i = 1 To {{Cl|UBOUND}}(handles) | ||
Line 129: | Line 127: | ||
EndOfFile = 0 | EndOfFile = 0 | ||
GET #handles(i), , s | {{Cl|GET (HTTP statement)|GET}} #handles(i), , s | ||
content(i) = content(i) + s | content(i) = content(i) + s | ||
Line 152: | Line 150: | ||
{{CodeEnd}} | {{CodeEnd}} | ||
---- | |||
;Example 4:This shows downloading straight to a file by providing the open file number. This approach is much more memory efficient if the HTTP response is large: | |||
{{CodeStart}} | {{CodeStart}} | ||
' Returns the status code of the HTTP response | ' Returns the status code of the HTTP response | ||
{{Cl|FUNCTION}} DownloadToFile&(url | {{Cl|FUNCTION}} DownloadToFile&(url {{Cl|AS}} {{Cl|STRING}}, fileHandle {{Cl|AS}} {{Cl|LONG}}) | ||
{{Cl|DIM}} h | {{Cl|DIM}} h {{Cl|AS}} {{Cl|LONG}}, content {{Cl|AS}} {{Cl|STRING}}, s {{Cl|AS}} {{Cl|STRING}} | ||
h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | h = {{Cl|_OPENCLIENT}}("HTTP:" + url) | ||
Line 165: | Line 164: | ||
{{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | {{Cl|WHILE}} {{Cl|NOT}} {{Cl|EOF}}(h) | ||
{{Cl|_LIMIT}} 60 | {{Cl|_LIMIT}} 60 | ||
GET #h, , s | {{Cl|GET (HTTP statement)|GET}} #h, , s | ||
{{Cl|PUT}} #fileHandle, , s | {{Cl|PUT}} #fileHandle, , s | ||
{{Cl|WEND}} | {{Cl|WEND}} | ||
Line 172: | Line 171: | ||
{{Cl|END FUNCTION}} | {{Cl|END FUNCTION}} | ||
{{CodeEnd}} | {{CodeEnd}} | ||
{{PageReferences}} |
Latest revision as of 13:15, 19 November 2024
QB64-PE starting in v3.5.0 includes built-in support for making HTTP requests and receiving their responses. This functionality is implemented in _OPENCLIENT, and the resulting HTTP handle can then be provided the various existing or new stream commands to interact with the response.
Before QB64-PE v4.0.0 HTTP functionality was unstable and requires $UNSTABLE:HTTP to be able to use it.
Starting the Connection
_OPENCLIENT is used to make an HTTP request, via using the HTTP: scheme. The string provided to _OPENCLIENT should be of the form "HTTP:url", where url is the HTTP url you want to make a request too. This can include anything valid in a URL, such as a port, resource path, query string, or fragment. The call to _OPENCLIENT will then return a handle to the HTTP connection which can be provided to all the rest of the commands related to HTTP requests. If the HTTP request cannot be made (for example, the domain is invalid) then zero is returned instead of a valid handle.
Metadata About the HTTP Response
The below commands all give metadata about the HTTP response. They're ordered in terms of usefulness:
- _STATUSCODE can be used to get the HTTP status code given in the HTTP response (Ex. 200, 404, 500, ...)
- LOF will return the 'Content Length' of the HTTP response if it was provided. It gives -1 if it is not known.
- EOF Will return whether the entire HTTP response has been read using GET.
- _CONNECTIONADDRESS$ can be used to get the "Effective URL" that was connected too, after taking redirects into account.
- _CONNECTED can be used to determine whether the HTTP connection to the server is still open.
In general for most HTTP responses you should ignore _CONNECTED and only make use of EOF. EOF will continue to return false until you have read all the data from the HTTP response and the connection has been closed, where-as _CONNECTED will return false even if you still have data to read.
Reading the HTTP Response Content
GET is used to read the data from the HTTP response. It should be used with a variable length string, which will then be resized by GET so that it contains all the data current available for the HTTP response. You can then either parse the data as it comes, or collect it all together to parse later. Data is available for reading with GET until EOF returns true. A _LIMIT or similar delay should be put in place when looping and using GET so that you don't use up too much CPU time. The actual downloading happens on a separate thread in your program, so how often you call GET has no impact on how fast the download is.
Closing The HTTP handle
Like any handle returned from _OPENCLIENT, all HTTP handles need to be closed by using CLOSE when you are finished with them.
Examples
- Example 1
- This function provides basic download functionality. The downloaded data will returned in a string, and the statuscode is provided so that you can check whether the request was successful.
' Content of the HTTP response is returned. ' The statusCode is also assigned. FUNCTION Download$(url AS STRING, statusCode AS LONG) DIM h AS LONG, content AS STRING, s AS STRING h = _OPENCLIENT("HTTP:" + url) statusCode = _STATUSCODE(h) WHILE NOT EOF(h) _LIMIT 60 GET #h, , s content = content + s WEND CLOSE #h Download$ = content END FUNCTION |
- Example 2
- This shows how you could modify the basic usage to provide a progress bar. Note that displaying progress is only possible if the server gives a `Content-length` header back in the response, which will be indicating to you via the result of LOF on the handle:
FUNCTION DownloadWithProgress$(url AS STRING, statusCode AS LONG) DIM h AS LONG, content AS STRING, s AS STRING, length AS LONG DIM progress AS DOUBLE h = _OPENCLIENT("HTTP:" + url) statusCode = _STATUSCODE(h) length = LOF(h) WHILE NOT EOF(h) _LIMIT 60 GET #h, , s content = content + s ' Display a progress bar if the Content-Length was provided IF length <> -1 THEN progress = CDBL(LEN(content)) / length LOCATE 1, 1 PRINT "["; PRINT STRING$(progress * 78, "*"); PRINT STRING$(78 - progress * 78, " "); PRINT "]"; ELSE LOCATE 1, 1 PRINT "[ Downloading..."; STRING$(80 - 17, " "); "]"; END IF WEND CLOSE #h Download$ = content END FUNCTION |
- Example 3
- This shows processing multiple downloads at the same time, with progress for each (Again keeping in mind that progress only works if LOF gives a valid length):
SUB DownloadMultipleWithProgress() DIM handles(1 To 3) AS LONG DIM content(1 To 3) AS STRING DIM length(1 to 3) AS LONG DIM i AS LONG, progress AS DOUBLE DIM EndOfFile AS LONG, s AS STRING handles(1) = _OPENCLIENT("HTTP:https://www.google.com") handles(2) = _OPENCLIENT("HTTP:https://www.google.com") handles(3) = _OPENCLIENT("HTTP:https://www.google.com") FOR i = 1 To UBOUND(handles) length(i) = LOF(handles(i)) NEXT DO _LIMIT 60 EndOfFile = -1 FOR i = 1 To UBOUND(handles) IF handles(i) = 0 THEN _CONTINUE IF EOF(handles(i)) THEN CLOSE #handles(i) handles(i) = 0 _CONTINUE END IF EndOfFile = 0 GET #handles(i), , s content(i) = content(i) + s ' Display a progress bar if the ' Content-Length was provided IF length <> -1 THEN progress = CDBL(LEN(content(i))) / length(i) LOCATE i, 1 PRINT "["; PRINT STRING$(progress * 78, "*"); PRINT STRING$(78 - progress * 78, " "); PRINT "]"; ELSE LOCATE i, 1 PRINT "[ Downloading..."; STRING$(80 - 17, " "); "]"; END IF NEXT LOOP While NOT EndOfFile ' The content() array now contains the results of the downloads END SUB |
- Example 4
- This shows downloading straight to a file by providing the open file number. This approach is much more memory efficient if the HTTP response is large:
' Returns the status code of the HTTP response FUNCTION DownloadToFile&(url AS STRING, fileHandle AS LONG) DIM h AS LONG, content AS STRING, s AS STRING h = _OPENCLIENT("HTTP:" + url) DownloadToFile& = _STATUSCODE(h) WHILE NOT EOF(h) _LIMIT 60 GET #h, , s PUT #fileHandle, , s WEND CLOSE #h END FUNCTION |