Uploading large files (1Meg) to S7 programmed as Embedded Web Server

Hello, 

I wrote an embedded web server program using the S7 and the NetX BSD module. I am able to serve web pages. The challenging part of my project is to use a web browser to get a file from the PC directories and upload it to the S7 and the file will be stored in an SD card. I have beginner java, javascript, html knowledge. From what I have researched so far, I used HTML form and file input to do the transfer. I was able to transfer a small text file. But when the file size became greater than 1166 bytes + HTTP headers = roughly 1460 bytes, the transfer fails. The 1460 makes sense because I gather that this is the maximum application layer data. Anyway, it looks like I am running into a memory issue, which means that I would have to send the files to the server in chunks. How do I do that in html? or in javascript? I also did more research and found out about PLUPLOAD. Would this work for my application? Can I embed this in a bwrowser? How? Are there other alternatives? Since i'm a beginner in all this, is there a simple Java applet  I could embed in the web browser that's easy to use for faster development time? Any suggestions on this is greatly appreciated.

Thanks.

AJ  

  • Have a look a t multi-part form data: developer.mozilla.org/.../FileUpDown

    Tim
  • Hi AJ,

    As Tim suggested, you may use multi-part form data and HTTP 1.1 streaming (Transfer-Encoding: chunked). Your server then can process incoming stream in chunks, for example once chunk of data is received, it can immediately write it to a file. This will allow the server not to buffer incoming data.

    Regards,
    adboc
  • In reply to Tim:

    Hi Tim and adboc,

    Thank you for responding. I went to the link and I have some questions.

    But before that, here is what I have done so far, I implemented a file transfer using HTML form with enctype = 'multipart/form-data'. Here is the code below

    <!DOCTYPE html>
    <html lang="en" dir="ltr">
    <head>
    <meta charset="utf-8">
    <title>File Transfer</title>
    <!------------------------>
    <style>
    body {
    width: 305px;
    margin: 0 auto;
    background-color: #ccc;
    }
    </style>
    <!------------------------>
    </head>
    <body>
    <h1>File Transfer</h1>
    <p><img src="First_In.jpg" alt="First-In"></p>
    <form method='POST' enctype='multipart/form-data' action='fup.cgi'>
    <p>Upload file</p>
    <input type="file" name="File" value="File">
    <p></p>
    <input type="submit" name="Upload" value="Upload">
    </form>
    </body>
    </html>

    With this I was able to transfer a small txt file (size less than 1166 bytes). But when it became greater than that, I could not transfer files anymore. I figured, I have some buffer/ memory issue on the server(S7) side. It looks like transferring in chunks is the way to go.

    The javascript code I saw on that link, will that automatically break the file into chunks? Or is it the same implementation as the HTML code above. Also, I have beginner html and javascript skills (I'm still currently working on javascript and html Udemy classes but my employer wants to see results ASAP). How do I use the function upload within the script tags?

    Also, abdoc, you mentioned about HTTP 1.1 streaming (Transfer-Encoding: chunked). I understand how to use this when I receive a GET command and responding back when I'm sending the web page data to the browser. This is how I'm sending huge images to appear on the browser. However, I don't understand how to implement this in the context of file transfer. Could you please elaborate more or point me to a direction or sample code on how I can implement this.

    Again, thank you very much for the help

    Regards,
    AJ
  • In reply to AJ:

    Hi AJ,

    Could you show the code that handles a file upload on Synergy side?

    Regards,
    adboc
  • In reply to adboc:

    Hi abdoc,

    I have not implemented the actual file upload but using BSD sockets I am able to get data from the browser and output it on a serial port. I attached 2 files, the BSD server socket implementation and the output I saw on the serial port for 2 cases, when the file transferred was small and the other was too big 

     

    Thanks

    AJ 

     

    Ouput.txt
    OUTPUT with small text file with "Hello"
    
    Message = POST /fup.cgi HTTP/1.1
    Referer: http://10.20.30.120/File_Transfer.html?File+Transfer=File+Transfer
    Cache-Control: max-age=0
    Content-Type: multipart/form-data; boundary=---------------------------7e259540702
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134
    Accept-Language: en-US
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Upgrade-Insecure-Requests: 1
    Accept-Encoding: gzip, deflate
    Host: 10.20.30.120
    Content-Length: 286
    Connection: Keep-Alive
    
    -----------------------------7e259540702
    Content-Disposition: form-data; name="File"; filename="Test.txt"
    Content-Type: text/plain
    
    Hello
    -----------------------------7e259540702
    Content-Disposition: form-data; name="Upload"
    
    Upload
    -----------------------------7e259540702--
    
    
    Number of bytes received = 866
    ===================================================================================================================
    
    
    OUTPUT when file too big, no contents is displayed
    
    Message = POST /fup.cgi HTTP/1.1
    Referer: http://10.20.30.120/File_Transfer.html?File+Transfer=File+Transfer
    Cache-Control: max-age=0
    Content-Type: multipart/form-data; boundary=---------------------------7e230c2040702
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134
    Accept-Language: en-US
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Upgrade-Insecure-Requests: 1
    Accept-Encoding: gzip, deflate
    Host: 10.20.30.120
    Content-Length: 12183
    Connection: Keep-Alive
    
    
    
    Number of bytes received = 584
    Server.txt
    #include "app_header.h"
    #include "web_server.h"
    /*********************************************/
    /* App Web Server Thread entry function */
    void app_web_server_thread_entry(void)
    {
        /* TODO: add your own code here */
        char val_array[10];
        int ret_val;
        unsigned int error_status;
        unsigned long actual_status;
        unsigned long local_ip_address, mask;
    
        int length;
        int ser_client_len;
        struct sockaddr_in ser_server_addr;
        struct sockaddr_in ser_client_addr;
    
        int opt = 1;
        int i, j;
        int sd;
        int max_sd;
        int activity;
        int new_socket;
        fd_set readfds;
        char err[10];
        int err_length;
        int *err_length_ptr;;
        int ser_num_bytes;
        int num_clients = 0;
    
        char *ip_string = STATIC_IP;
        /*********************************************/
        tx_semaphore_get(&g_start_enet_semaphore, TX_WAIT_FOREVER);
        nx_ip_address_set(&g_ip0, inet_addr(ip_string), inet_addr("255.255.255.0"));
        write_console_msg("\n\n\rChecking for ethernet link...");
        do
        {
            /* Make sure we have a valid network link. */
            error_status =  nx_ip_status_check(&g_ip0, NX_IP_LINK_ENABLED, &actual_status, 100);
        } while (error_status != NX_SUCCESS);
        write_console_msg("\n\rEthernet status ok");
        /*********************************************/
        //Obtain local static IP (IP is set in the module properties)
        ret_val = (INT)nx_ip_address_get(&g_ip0, &local_ip_address, &mask);
        write_console_msg("\n\rLocal Static IP = ");
        display_ip_address(local_ip_address);
        Cli_State.Client_Local_IP = local_ip_address;
        /*********************************************/
        write_console_msg("\n\n\rInitializing web server master socket...");
        /*********************************************/
        /*********************************************/
        for (i=0; i<MAX_CLIENTS; i++)
        {
            Ser_State[i].Server_Sock_ID = 0;
            Ser_State[i].Server_Status = 0;
            Ser_State[i].Server_Error_Status = 0;
            Ser_State[i].Server_Num_Bytes = 0;
            Ser_State[i].Server_Rcv_Index = 0;
        }
        /*********************************************/
        length = sizeof(struct sockaddr_in);
        ser_client_len = sizeof(ser_client_addr);
        memset(&ser_server_addr, 0, sizeof(ser_server_addr));
        ser_server_addr.sin_family = AF_INET;
        ser_server_addr.sin_addr.s_addr = htonl(local_ip_address);
        ser_server_addr.sin_port = htons(80);
        /*********************************************/
        Master_Socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
        write_console_msg("\n\rMaster_Socket = ");
        long_to_ascii(Master_Socket, val_array);
        write_console_msg(val_array);
        if(Master_Socket == 0)
        {
            write_console_msg("\n\rMaster socket failed");
        }
        /*********************************************/
        ret_val = setsockopt(Master_Socket, SOL_SOCKET, SO_REUSEADDR | SO_ERROR, (char *)&opt, sizeof(opt));
        if(ret_val < 0)
        {
            write_console_msg("\n\rError setsockopt");
        }
        /*********************************************/
        ret_val = bind(Master_Socket, (struct sockaddr *) &ser_server_addr, sizeof(ser_server_addr));
        if(ret_val < 0)
        {
            write_console_msg("\n\rError bind ");
        }
        /*********************************************/
        ret_val = listen(Master_Socket, 5);
        if(ret_val < 0)
        {
            write_console_msg("\n\rError listen");
        }
        /*********************************************/
        Number_Of_Objects = 0;
        Total_Objects = 1;
        err_length = 10;
        err_length_ptr = &err_length;
        /*********************************************/
        while(1)
        {
            write_console_msg("\n\n\rLOOP START");
            //clear the socket set
            FD_ZERO(&readfds);
    
            //add master socket to set
            FD_SET(Master_Socket, &readfds);
            max_sd = Master_Socket;
    
            write_console_msg("\n\n\rNo. of clients open = ");
            long_to_ascii(num_clients, val_array);
            write_console_msg(val_array);
    
            //add child sockets to set
            for (i=0 ; i<MAX_CLIENTS ; i++)
            {
                //socket descriptor
                sd = Ser_State[i].Server_Sock_ID;
    
                //if valid socket descriptor then add to read list
                if(sd > 0)
                {
                    FD_SET(sd , &readfds);
                }
                //highest file descriptor number, need it for the select function
                if(sd > max_sd)
                {
                    max_sd = sd;
                }
            }//end for (i=0 ; i<MAX_CLIENTS ; i++)
    
            write_console_msg("\n\n\rWaiting for client activity...");
            //wait for an activity on one of the sockets , timeout is NULL , so wait indefinitely
            activity = select( max_sd + 1 , &readfds , NULL , NULL , NULL);
            if((activity < 0) && (errno!=EINTR))
            {
                write_console_msg("\n\rError select");
            }//end if((activity < 0) && (errno!=EINTR))
    
            //If something happened on the master socket , then its an incoming connection
            if(FD_ISSET(Master_Socket, &readfds))
            {
                new_socket = accept(Master_Socket, (struct sockaddr *)&ser_client_addr, &ser_client_len);
                if (new_socket <= 0)
                {
                    write_console_msg("\n\rNew socket failed");
                    ret_val = getsockopt(Master_Socket, SOL_SOCKET, SO_ERROR, (char *)&err, err_length_ptr);
                    if(ret_val < 0)
                    {
                        write_console_msg("\n\rError setsockopt");
                    }
                    else
                    {
                        write_console_msg(err);
                    }
                    for (i=0; i<MAX_CLIENTS; i++)
                    {
                        //if position is empty
                        if(Ser_State[i].Server_Sock_ID > 0)
                        {
                            sd = Ser_State[i].Server_Sock_ID;
                            num_clients--;
                            write_console_msg("\n\rClient disconnected...");
                            write_console_msg("\n\rSocket ID = ");
                            long_to_ascii(sd, val_array);
                            write_console_msg(val_array);
                            getpeername(sd, (struct sockaddr *) &ser_client_addr, &length);
                            write_console_msg("\n\rServer app remote port = ");
                            ulong_to_ascii(ser_client_addr.sin_port, val_array);
                            write_console_msg(val_array);
                            write_console_msg("\n\rServer app remote IP = ");
                            display_ip_address(ser_client_addr.sin_addr.s_addr);
    
                            //Close the socket and mark as 0 in list for reuse
                            Ser_State[i].Server_Status = 0;
                            Ser_State[i].Server_Sock_ID = 0;
                            write_console_msg("\n\rServer socket closed from server receive");
                            soc_close(Ser_State[i].Server_Sock_ID);
                        }//end if(Ser_State[i].Server_Sock_ID == 0)
                    }//end for (i=0; i<MAX_CLIENTS; i++)
                    soc_close(Master_Socket);
                    /*********************************************/
                    /*********************************************/
                    Master_Socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
                    write_console_msg("\n\rMaster_Socket = ");
                    long_to_ascii(Master_Socket, val_array);
                    write_console_msg(val_array);
                    if(Master_Socket == 0)
                    {
                        write_console_msg("\n\rMaster socket failed");
                    }
                    /*********************************************/
                    ret_val = setsockopt(Master_Socket, SOL_SOCKET, SO_REUSEADDR | SO_ERROR, (char *)&opt, sizeof(opt));
                    if(ret_val < 0)
                    {
                        write_console_msg("\n\rError setsockopt");
                    }
                    /*********************************************/
                    ret_val = bind(Master_Socket, (struct sockaddr *) &ser_server_addr, sizeof(ser_server_addr));
                    if(ret_val < 0)
                    {
                        write_console_msg("\n\rError bind ");
                    }
                    /*********************************************/
                    ret_val = listen(Master_Socket, 5);
                    if(ret_val < 0)
                    {
                        write_console_msg("\n\rError listen");
                    }
                    /*********************************************/
                    continue;
                }
                /*********************************************/
                else
                {
                    //inform user of socket number - used in send and receive commands
                    write_console_msg("\n\rNew connection: ");
                    write_console_msg("\n\rSocket ID = ");
                    long_to_ascii(new_socket, val_array);
                    write_console_msg(val_array);
                    //Display local info
                    getsockname(new_socket, (struct sockaddr *)&ser_server_addr, &length);
                    write_console_msg("\n\rServer app local port = ");
                    ulong_to_ascii(ser_server_addr.sin_port, val_array);
                    write_console_msg(val_array);
                    write_console_msg("\n\rServer app local IP = ");
                    display_ip_address(ser_server_addr.sin_addr.s_addr);
                    /*********************************************/
                    //Display remote client info
                    getpeername(new_socket, (struct sockaddr *) &ser_client_addr, &length);
                    write_console_msg("\n\rServer app remote port = ");
                    ulong_to_ascii(ser_client_addr.sin_port, val_array);
                    write_console_msg(val_array);
                    write_console_msg("\n\rServer app remote IP = ");
                    display_ip_address(ser_client_addr.sin_addr.s_addr);
                    /*********************************************/
                    //add new socket to array of sockets
                    for (i=0; i<MAX_CLIENTS; i++)
                    {
                        //if position is empty
                        if(Ser_State[i].Server_Sock_ID == 0)
                        {
                            num_clients++;
                            Ser_State[i].Server_Status |= SER_CONNECTED;
                            Ser_State[i].Server_Sock_ID = new_socket;
                            write_console_msg("\n\rAdding to list of sockets as ");
                            ulong_to_ascii((unsigned int)i, val_array);
                            write_console_msg(val_array);
                            break;
                        }//end if(Ser_State[i].Server_Sock_ID == 0)
                    }//end
                }//end else
            }//end if (FD_ISSET(Master_Socket, &readfds))
            //else its some IO operation on some other socket
            for (i=0; i<MAX_CLIENTS; i++)
            {
                sd = Ser_State[i].Server_Sock_ID;
                if(FD_ISSET(sd , &readfds))
                {
                    //Check if it was for closing , and also read the incoming message
                    ser_num_bytes = recv(Ser_State[i].Server_Sock_ID, (VOID *)Ser_State[i].Server_Rcv_Buff, 8192, 0);
                    if(ser_num_bytes <= 0)
                    {
                        //Somebody disconnected , get his details and print
                        num_clients--;
                        write_console_msg("\n\n\rClient disconnected...");
                        write_console_msg("\n\rSocket ID = ");
                        long_to_ascii(sd, val_array);
                        write_console_msg(val_array);
                        getpeername(sd, (struct sockaddr *) &ser_client_addr, &length);
                        write_console_msg("\n\rServer app remote port = ");
                        ulong_to_ascii(ser_client_addr.sin_port, val_array);
                        write_console_msg(val_array);
                        write_console_msg("\n\rServer app remote IP = ");
                        display_ip_address(ser_client_addr.sin_addr.s_addr);
    
                        ret_val = soc_close(Ser_State[i].Server_Sock_ID);
                        if(ret_val < 0)
                        {
                            write_console_msg("\n\rError close");
                        }
                        else
                        {
                            //Close the socket and mark as 0 in list for reuse
                            Ser_State[i].Server_Status = 0;
                            Ser_State[i].Server_Sock_ID = 0;
                            write_console_msg("\n\rServer socket closed from server receive");
                        }
                    }//end if(ser_num_bytes == 0)
                    else
                    {
                        write_console_msg("\n\n\rMessage from client...");
                        write_console_msg("\n\rSocket ID = ");
                        long_to_ascii(sd, val_array);
                        write_console_msg(val_array);
    
                        Ser_State[0].Server_Status |= SER_RECEIVING;
                        write_console_msg("\n\rMessage = ");
                        for(j=0; j<ser_num_bytes; j++)
                        {
                            write_console_char((char)Ser_State[i].Server_Rcv_Buff[j]);  ------------------>>This is the output to serial port
                        }//end for(i=0; i<Cli_State.Client_Num_Bytes; i++)
    
                        write_console_msg("\n\n\rNumber of bytes received = ");
                        long_to_ascii(ser_num_bytes, val_array);
                        write_console_msg(val_array);
                        write_console_msg("\n\n\rWeb server message received = ");
    
                        Ser_State[i].Server_Status &= ~SER_RECEIVING;
                        Ser_State[i].Server_Rcv_Index = Ser_State[i].Server_Num_Bytes;
    
                        run_web_server(i); -------------------> this function takes the browser command like GET and it sends the webpages to the browser
                                                                 I have not implemented anything for POST yet
                    }//end else
                }//end if(FD_ISSET(sd , &readfds))
            }//end for (i=0; i<MAX_CLIENTS; i++)
        }//end while(1)
    }//end void app_web_server_thread_entry(void)
    

  • In reply to AJ:

    By the way, when I said I have not implemented the file upload, I meant I have not done anything to store it anywhere but I am able to read it and display it straight to serial port. But as you can see, when the text file became to large, no contents were displayed. And just a note, I also used the HTML code I posted previously to do the file transfer using "form" and "file input".
  • In reply to AJ:

    Hi AJ,

    Based on an example from NetX Duo BSD user's guide, I built a simple Synergy application for SK-S7G2. Then tried to upload 2200 bytes file and here is the output from the console:

    Received 1460 bytes
    Received 1366 bytes
    Received 2826 bytes in total

    Which seems to be correct (2200 + additional bytes for HTTP metadata). The project is attached below:

    S7G2_SK_NetX_BSD_1_5_0.zip

    Regards,
    adboc

  • In reply to adboc:

    Hi Adboc,

    Thanks. I was finally able to upload data using XMLHttp on javascript and breaking the file into chunks. This link helped a lot

    www.accelebrate.com/.../

    The second part is about Ajax and using the code the author provided, I tweaked it for my application.

    Regards,
    AJ