File downloading [ SOLVED ]

25 posts / 0 new
Last post
Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
File downloading [ SOLVED ]

Hello,
in my application I need to download (and then process) a file hosted at a given URL (for example: http://www.domain.com/screenshot.jpg).
How to achieve this?
The target is AmigaOS 4.1 Update 1.

Thanks, regards

zzd10h
zzd10h's picture
Offline
Last seen: 6 years 4 months ago
Joined: 2012-08-24 20:56
And you don't want to use

And you don't want to use wget ? (ugly method)

trixie
trixie's picture
Offline
Last seen: 4 months 4 weeks ago
Joined: 2011-02-03 13:58
@massi Use libcurl, it's on

@massi

Use libcurl, it's on OS4depot:

http://www.os4depot.net/index.php?function=showfile&file=development/library/misc/libcurl.lha

AmigaOne X5000-020 / 2GB RAM / Sapphire Pulse Radeon RX 560 / AmigaOS 4.1 Final Edition Update 2

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@zzd10h I prefer a clean

@zzd10h

I prefer a clean solution.

@trixie

Can it be done in a native way, without additional dependencies?

trixie
trixie's picture
Offline
Last seen: 4 months 4 weeks ago
Joined: 2011-02-03 13:58
@massi It's compiled for OS4

@massi

It's compiled for OS4 - how more native would you like it? :-) If you meant whether the OS provides a function for downloading files then the answer is no, AFAIK.

And it's a link library, not a shared library or object, so there are no dependencies.

AmigaOne X5000-020 / 2GB RAM / Sapphire Pulse Radeon RX 560 / AmigaOS 4.1 Final Edition Update 2

kas1e
kas1e's picture
Offline
Last seen: 1 year 5 months ago
Joined: 2010-11-30 15:30
@massi Or you can go

@massi

Or you can go very-native-oldshool-way, and write everything from scratch on plain C: handle ip/tcp packets/structures/connects for youself, and download the file as binary data from whatever you need via recv() kind functions. I.e. netinet/in.h, ip.h, maybe tcp.h and co, depends of how deep you want to jump into the roots.

Buld tcp/ip connection like this:

  1. sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  2.  
  3. cli.sin_family = AF_INET;
  4. cli.sin_addr.s_addr = ip;
  5. cli.sin_port = port;
  6.  
  7. if (connect(sock, (struct sockaddr *) &cli,
  8. sizeof(cli)) < 0) {
  9. close(sock);
  10. printf("No connect\n");
  11. exit(0);
  12. }

after you connect, recieve the data to buffer via recvfrom() or recv(), and write to local file , then close socket and co. Of course if server to which you will connect are not your one (which after connection just will send data), but plain http, then you will need to deal with http packets as well manually (so RFC on http protocol, to see the bits you need and co). In general all what you will need to do, its just do some http protocol related stuff, and recieve answer (your data).

http protocol are pretty simply (https support will be another deal, because there as far as i remember openssl will be involved), and it can be enough to do something like this:

  1. if(getsockopt(sockets[i].s, SOL_SOCKET, SO_ERROR, &error, &err_len) < 0 || error != 0)
  2. {
  3. shutdown(sockets[i].s,2);
  4. close(sockets[i].s);
  5. sockets[i].state=0;
  6. }
  7. else
  8. write(sockets[i].s,"GET /patch/to_file_for_download/file.zip HTTP/1.0\x0d\x0a\x0d\x0a",54);

And answer will be your data.

I.e. you make a ip / tcp connection yourself from scratch via ip structs, and deals with http.

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@trixie OK thanks, it can be

@trixie

OK thanks, it can be a way.

@kas1e

I always like the from "scratch" approach :) thanks.

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
Perhaps it can be achieved

Perhaps it can be achieved through a dedicated AmigaDOS handler / device (HTTP:, ...)?
Anyone aware of this?
Any already existing solutions?

salass00
salass00's picture
Offline
Last seen: 1 year 1 month ago
Joined: 2011-02-03 11:27
@Massi Perhaps it can be

@Massi


Perhaps it can be achieved through a dedicated AmigaDOS handler / device (HTTP:, ...)?
Anyone aware of this?
Any already existing solutions?

There is a http-handler from Chris Young that works pretty well:
http://aminet.net/package/comm/www/httphandler

thomas
thomas's picture
Offline
Last seen: 3 hours 8 min ago
Joined: 2011-05-16 14:23
Why is this less ugly than

Why is this less ugly than the wget approach?

IMHO it is even more ugly because wget is a simple program you can include in your program's drawer, but the http handler needs installation into the user's system directories.

Why don't you just search Google for a simple httpget example and use this in your code?

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@salass00,

@salass00, @thomas

Thanks.
At this stage I am evaluating possible solutions. Overall I prefer Amiga styled techniques.

PJS
PJS's picture
Offline
Last seen: 4 years 2 months ago
Joined: 2012-12-09 18:38
@massi FYI, Roadshow does

@massi

FYI, Roadshow does provide a simple handler that you can use, the TCP: device.

You can access online content just like reading/writing a file with a "TCP:[url]/[port]/..."
syntax. I've used it a fair bit in ARexx both for sending/getting data.

For more info, see:

SYS:Documentation/Roadshow/tcp-handler.doc

Then again, I've been told this isn't the best way to do the fastest or most reliable
transfers. For that, you can use kas1e's low level, longhand approach.

Good luck,

PJS

kas1e
kas1e's picture
Offline
Last seen: 1 year 5 months ago
Joined: 2010-11-30 15:30
@massi With tcp/ip, "native

@massi
With tcp/ip, "native amiga way" will be the same as any bsd/linux one, just you can use bsdsocket.library (which make no big sense in compare with newlib sockets).

All that downloading of files on lowlevel are discussed a lot by all the hackers for last 20 years everythere , so Thomas's suggestion about getting any gethttp.c from google will almost do all what you need, just some adoptation and understanding will need it.

You can even go deeper if you want real hardcore: once you write a C code for httpget, you can translate that all to the ppc assembler with removing un-necessary parts and optimise it (as most of those hackers do when do shellcodes). For example check x86/linux realisation here. You can see how small is it (and C source is no much bigger), for downloading a file from http and executing it. And as our RoadShow its just almost the same BSD tcp/ip stack, its all will be kind of the same by logic, just different assembler. I.e. will be the same small and lowlevel.

I even found one more example in google which i compile and test on os4 without changes (except that i change ip on os4coding.net's one, and compile it with -D__USE_INLINE__):

  1. #include <sys/types.h>
  2. #include <sys/socket.h>
  3. #include <netinet/in.h>
  4. #include <string.h>
  5. #include <stdio.h>
  6. #include <stdlib.h>
  7.  
  8. int main ( void ) {
  9. int socket_handle ;
  10. struct sockaddr_in socket_detials ;
  11. char * input_buffer;
  12. char * httpget = "GET HTTP 1.1 / \x0D\x0A\n\x0D\x0A\n" ;
  13.  
  14. input_buffer = malloc(20000);
  15.  
  16. socket_handle = socket ( AF_INET, SOCK_STREAM, 0) ;
  17. socket_detials.sin_family = AF_INET ;
  18. socket_detials.sin_addr.s_addr=inet_addr("68.90.68.66");
  19. socket_detials.sin_port = htons(80);
  20. bzero ( &(socket_detials.sin_zero), 8 ) ;
  21.  
  22. if ( connect (socket_handle,(struct sockaddr*)&socket_detials, sizeof ( struct sockaddr)) == -1 ){
  23. printf ( "Couldnt connect to server\n" ) ;
  24. }
  25. printf ( "Sending %d bytes\n", send ( socket_handle , httpget, strlen(httpget), 0 ) ) ;
  26. printf ( "Received %d bytes\n", recv ( socket_handle , input_buffer , 20000, 0 ) ) ;
  27. printf ( "%s\n", input_buffer ) ;
  28.  
  29. return 0 ;
  30. }

And there is result:

  1. 6/0.Work:code_examples> gcc -D__USE_INLINE__ httpget2.c -o httpget
  1. 6/0.Work:code_examples> httpget
  1. Sending 21 bytes
  2. Received 1360 bytes
  3. HTTP/1.1 400 Bad Request
  4. Date: Sun, 09 Dec 2012 17:45:40 GMT
  5. Server: Apache/2.2.16 (Debian)
  6. Last-Modified: Tue, 23 Oct 2012 05:33:02 GMT
  7. ETag: "2ca015-700-4ccb34db89f80"
  8. Accept-Ranges: bytes
  9. Content-Length: 1792
  10. Vary: Accept-Encoding
  11. Connection: close
  12. Content-Type: text/html
  13.  
  14. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  15. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  16. <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  17. <head>
  18. <title>ERROR 400 - Bad Request!</title>
  19. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  20. <meta name="robots" content="noindex" />
  21. <style type="text/css"><!--
  22. body {
  23. color: #444444;
  24. background-color: #EEEEEE;
  25. font-family: 'Trebuchet MS', sans-serif;
  26. font-size: 80%;
  27. }
  28. h1 {}
  29. h2 { font-size: 1.2em; }
  30. #page{
  31. background-color: #FFFFFF;
  32. width: 60%;
  33. margin: 24px auto;
  34. padding: 12px;
  35. }
  36. #header {
  37. padding: 6px ;
  38. text-align: center;
  39. }
  40. .status3xx { background-color: #475076; color: #FFFFFF; }
  41. .status4xx { background-color: #C55042; color: #FFFFFF; }
  42. .status5xx { background-color: #F2E81A; color: #000000; }
  43. #content {
  44. padding: 4px 0 24px 0;
  45. }
  46. #footer {
  47. color: #666666;
  48. background:
  49.  
  50. 6/0.Work:code_examples>

I.e. its already works , and get from os4coding http answer. Now you just need to add code for filling correct ip/domen and path to file from whatevewer you need, and save answer to file. Then you can compile it with -S and -O2, and make a real hardcore on asm :)

ps. Blah, that source even from amiga.org archives, there is full thread with details, and Piru's example is much better of course

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@PJS Thanks, AmigaDOS

@PJS

Thanks, AmigaDOS handlers (HTTP:, TCP:) are definitely what I was looking for.

@kas1e

Thanks, very valuable contribution and really hacker style which I always appreciate ;)

thomas
thomas's picture
Offline
Last seen: 3 hours 8 min ago
Joined: 2011-05-16 14:23
Here is a Rexx program which

Here is a Rexx program which downloads a file using the TCP handler:

  1. /* rexx */
  2.  
  3. parse arg url
  4.  
  5. if url = "" then url = "http://www.os4coding.net/sites/default/files/interbase_light_logo.png"
  6.  
  7. parse var url prot "://" server "/" path
  8.  
  9. if prot ~= "http" then do
  10. say "Only http URLs are supported,"
  11. exit 20
  12. end
  13.  
  14. parse var server server ":" port
  15.  
  16. if server = "" then do
  17. say "invalid URL."
  18. exit 20
  19. end
  20.  
  21. if port = "" then port = 80
  22.  
  23. path = "/"path
  24.  
  25. x = lastpos("/",path)
  26. file = translate(substr(path,x+1),"________",":#?()[] ")
  27.  
  28. if file = "" then file = "index.html"
  29.  
  30. file = "RAM:"file
  31.  
  32. say "server = <"server">"
  33. say "port = <"port">"
  34. say "path = <"path">"
  35. say "file = <"file">"
  36. say ""
  37.  
  38. cr = '0d'x
  39. ok = 0
  40.  
  41. if Open(tcp,"tcp:"server"/"port,write) then do
  42. call WriteLn tcp,"GET" path "HTTP/1.1"cr
  43. call WriteLn tcp,"Host:" server||cr
  44. call WriteLn tcp,cr
  45. maxsize = 0
  46. l = strip(ReadLn(tcp),'T',cr)
  47. do while (l ~= "")
  48. say l
  49. if word(l,1) = "Content-Length:" then maxsize = word(l,2)
  50. l = strip(ReadLn(tcp),'T',cr)
  51. end
  52. if Open(out,file,write) then do
  53. size = 0
  54. do while (~eof(tcp))
  55. buffer = ReadCh(tcp,1024)
  56. len = length(buffer)
  57. size = size + len
  58. call WriteCh out,buffer
  59. if maxsize > 0 & size >= maxsize then leave
  60. end
  61. call Close(out)
  62. say ""
  63. say "wrote" size "bytes to" file
  64. ok = 1
  65. end
  66. else
  67. say "cannot open output file" file
  68. call Close(tcp)
  69. end
  70. else
  71. say "cannot connect to" server "port" port
  72.  
  73. if ok then
  74. address command 'multiview "'file'"'
Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@thomas Thanks for this good

@thomas

Thanks for this good start, it should be straight to adapt it to C language.

afxgroup
afxgroup's picture
Offline
Last seen: 10 months 2 weeks ago
Joined: 2011-02-03 15:26
All those solutions doesn't

All those solutions doesn't handle web server response like "moved" or other kind of response.. the best was to use libcurl that can handle everything.. or wget

TSK
TSK's picture
Offline
Last seen: 5 months 3 weeks ago
Joined: 2011-06-28 02:06
@thomas Thanks big times ! I

@thomas

Thanks big times ! I got it finally how to use this TCP: device.

Hans
Hans's picture
Offline
Last seen: 2 months 3 weeks ago
Joined: 2010-12-09 22:04
@Massi I'll add my voice to

@Massi

I'll add my voice to those suggesting using LibCURL. If you were to write something from scratch, you'll effectively end up writing your own version of LibCURL's HTTP handling code. AmigaOS' networking API is the same BSD sockets that all major OSes uses, and LibCurl calls the BSD socket functions directly. So, LibCURL's code is about as "native" as you can get on all of the platforms that it supports.

There is no real reason to write your own from scratch, unless you're doing it for the learning experience.

Hans

http://hdrlab.org.nz/ - Amiga OS 4 projects, programming articles and more. Home of the RadeonHD driver for Amiga OS 4.x project.
http://keasigmadelta.co.nz/ - more of my software.

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@Hans I agree with you,

@Hans

I agree with you, there is no need to "reinvent the wheel" each time ...
I personally prefer AmigaDOS handlers over other solutions because they are the Amiga way of doing certain things.
Good to list and talk about different techniques anyway.

hypex
hypex's picture
Offline
Last seen: 1 month 2 weeks ago
Joined: 2011-09-09 16:20
@thomas Why is this less

@thomas

Why is this less ugly than the wget approach?

Isn't it obvious? ;-)

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
@thomas Does the Rexx script

@thomas

Does the Rexx script work with HTTP 1.1 chunked transfer mode (response arriving in multiple parts)?

Perhaps chunked messages are managed within the TCP: handler or?

thomas
thomas's picture
Offline
Last seen: 3 hours 8 min ago
Joined: 2011-05-16 14:23
The TCP handler only opens a

The TCP handler only opens a simple TCP (telnet) connection. It does not know HTTP or any other protocol. If you want to support chunked transfers, you need to program that. The example only shows a very simple "get that small file from my own server", for example to check for a new version of your software. It's obviously not a complete implementation of all features of the HTTP protocol.

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
Thanks to all, I now have a

Thanks to all, I now have a working example, it is written in C language and it makes use of the TCP: handler. A bit of refining is needed anyway.
Going forward in my project.

Massi
Massi's picture
Offline
Last seen: 3 years 10 months ago
Joined: 2012-03-28 17:16
For those interested, I have

For those interested, I have finalized this topic to a complete application.
The project name is BigSister, available from OS4 Depot, source code included in the package.

Regards

Log in or register to post comments