Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Efficient File Uploads with JavaScript and PHP: A Chunking Approach

Tech May 8 3

Handling large file uploads in web applications can be challenging due to potential timeouts and buffer limitations. A common and robust solution involves breaking down the file into smaller chunks on the client-side and uploading them sequentially to the server.

Core Concepts:

  • File API: Modern browsers provide the File API, which allows JavaScript to interact with files selected by the user via an input type="file" element. The FileList object, accessible from the input's files property, represents the selected files.
  • FormData: The FormData object is used to construct a set of key/value pairs representing form fields and their values. It's ideal for sending data, including files, using asynchronous requests (like XMLHttpRequest or fetch), especially when the form's enctype is set to multipart/form-data.
  • Blob: A Blob (Binary Large Object) represents raw, immutable data, similar to a file. The File interface is built upon Blob, adding file-specific properties. The crucial method here is Blob.prototype.slice(), which extracts a portion of the Blob as a new Blob object.

Chunking Process:

  1. File Slicing: The Blob.slice(start, end) method is used to divide the selected file into manageable chunks. The start and end parameters define the byte range for each chunk.
  2. Sending Chunks: Each file chunk is packaged into a FormData object, along with metadata such as the original filename, total chunk count, current chunk number, and file size. These FormData objects are then sent to the server, typically via XMLHttpRequest.
  3. Server-Side Assembly: The PHP backend receives each chunk. It can create a temporary directory for the upload or use a naming convension to store chunks. As chunks arrive, they are appended to a temporary file on the server.
  4. Final Assembly: Once the server has received all chunks for a given file (indicated by the current chunk number matching the total chunk count), it can assemble the complete file. The temporary chunks can then be merged into the final file, and any temporary storage can be cleaned up.

Implementation Example:

Front end (JavaScript):


class AjaxSender {
    constructor(method, url, requestData) {
        this.method = method;
        this.url = url;
        this.requestData = requestData;
    }

    // Helper to format data for non-FormData requests
    formatRequestData() {
        let dataString = '';
        const keys = Object.keys(this.requestData);
        const values = Object.values(this.requestData);
        for (let i = 0; i < keys.length; i++) {
            dataString += `${keys[i]}=${values[i]}&`;
        }
        return dataString.slice(0, -1); // Remove trailing '&'
    }

    send(onSuccess) {
        const xhr = new XMLHttpRequest();
        xhr.onreadystatechange = function() {
            if (xhr.readyState === 4) {
                if (xhr.status === 200) {
                    onSuccess(xhr.response);
                } else {
                    console.error('AJAX request failed:', xhr.status, xhr.statusText);
                }
            }
        };

        if (this.method.toUpperCase() === "POST") {
            xhr.open("POST", this.url);
            // Set Content-Type only if not sending FormData
            if (!(this.requestData instanceof FormData)) {
                 xhr.setRequestHeader('content-type', 'application/x-www-form-urlencoded');
            }
            xhr.send(this.requestData instanceof FormData ? this.requestData : this.formatRequestData());
        } else if (this.method.toUpperCase() === "GET") {
            xhr.open('GET', `${this.url}?${this.formatRequestData()}`);
            xhr.send();
        }
    }
}

class FileChunker {
    constructor(file, chunkSize = 1 * 1024 * 1024) { // Default chunk size: 1MB
        if (!(file instanceof File)) {
            throw new Error("Invalid input: Expected a File object.");
        }
        this.file = file;
        this.chunkSize = chunkSize;
    }

    // Splits the file into chunks
    split() {
        const fileSize = this.file.size;
        const fileName = this.file.name;
        const chunks = [];
        let start = 0;

        while (start < fileSize) {
            const end = Math.min(start + this.chunkSize, fileSize);
            const chunk = this.file.slice(start, end);
            const formData = new FormData();

            formData.append("fileChunk", chunk);
            formData.append("fileName", fileName);
            formData.append("fileSize", this.file.size); // Total file size
            formData.append("chunkIndex", chunks.length + 1); // 1-based index
            formData.append("totalChunks", Math.ceil(fileSize / this.chunkSize));
            formData.append("uploadType", "chunk"); // Custom type identifier

            chunks.push(formData);
            start = end;
        }
        return chunks;
    }
}

class ChunkUploader {
    constructor(uploadUrl, fileChunks) {
        this.uploadUrl = uploadUrl;
        if (!Array.isArray(fileChunks)) {
            throw new Error("Invalid input: Expected an array of FormData chunks.");
        }
        this.fileChunks = fileChunks;
        this.currentChunkIndex = 0;
    }

    sendNextChunk() {
        if (this.currentChunkIndex >= this.fileChunks.length) {
            console.log("All chunks uploaded.");
            return;
        }

        const currentFormData = this.fileChunks[this.currentChunkIndex];
        const chunkMetadata = {
            fileName: currentFormData.get('fileName'),
            fileSize: currentFormData.get('fileSize'),
            chunkIndex: parseInt(currentFormData.get('chunkIndex')),
            totalChunks: parseInt(currentFormData.get('totalChunks'))
        };

        const xhr = new XMLHttpRequest();
        xhr.onreadystatechange = () => {
            if (xhr.readyState === 4) {
                if (xhr.status === 200) {
                    const response = JSON.parse(xhr.responseText);
                    console.log(`Chunk ${chunkMetadata.chunkIndex}/${chunkMetadata.totalChunks} uploaded successfully:`, response);

                    if (response.status === 200 && response.message === "Chunk upload successful") {
                        this.currentChunkIndex++;
                        // Use setTimeout to avoid blocking the main thread and allow UI updates
                        setTimeout(() => this.sendNextChunk(), 0);
                    } else if (response.status === 200 && response.message === "File upload complete") {
                        console.log("File uploaded successfully! URL:", response.fileUrl);
                        // Handle successful completion (e.g., display message to user)
                    } else {
                        console.error("Upload error for chunk:", chunkMetadata.chunkIndex, response);
                        // Handle specific errors or retry logic
                    }
                } else {
                    console.error(`Error uploading chunk ${chunkMetadata.chunkIndex}:`, xhr.status, xhr.statusText);
                    // Implement retry mechanism or user notification
                }
            }
        };

        xhr.open("POST", this.uploadUrl, true); // true for asynchronous
        xhr.send(currentFormData);
    }

    startUpload() {
        if (this.fileChunks.length === 0) {
            console.warn("No file chunks to upload.");
            return;
        }
        this.sendNextChunk();
    }
}

// --- Event Listener Example ---
const fileInput = document.getElementById('fileInput'); // Assuming an input with id="fileInput"
const uploadEndpoint = 'http://your-server.com/upload.php'; // Replace with your actual PHP endpoint

fileInput.addEventListener('change', function(event) {
    if (event.target.files.length > 0) {
        const selectedFile = event.target.files[0];
        try {
            const chunker = new FileChunker(selectedFile);
            const chunks = chunker.split();
            const uploader = new ChunkUploader(uploadEndpoint, chunks);
            uploader.startUpload();
        } catch (error) {
            console.error("Error during file processing:", error);
            // Display error message to the user
        }
    }
});

Backend (PHP):

<?php
header(
 500, 'message' => 'Internal Server Error'];

    // Basic validation
    if (!isset($_FILES['fileChunk']) || !isset($_POST['fileName']) || !isset($_POST['chunkIndex']) || !isset($_POST['totalChunks'])) {
        $response = ['status' => 400, 'message' => 'Bad Request: Missing parameters.'];
    } else {
        $chunkFile = $_FILES['fileChunk'];
        $fileName = basename($_POST['fileName']); // Sanitize filename
        $chunkIndex = (int)$_POST['chunkIndex'];
        $totalChunks = (int)$_POST['totalChunks'];

        // Define the final file path
        $finalFilePath = FileHandler::UPLOAD_DIR . $fileName;
        $tempChunkPath = $chunkFile['tmp_name'];

        // Append the current chunk to the final file
        if (FileHandler::appendChunkToFile($finalFilePath, $tempChunkPath)) {
            // Check if this is the last chunk
            if ($chunkIndex === $totalChunks) {
                // File upload is complete
                $response = [
                    'status' => 200,
                    'message' => 'File upload complete',
                    'fileUrl' => '/uploads/' . $fileName // Relative URL for the client
                ];
                 // Optional: Perform post-upload processing, e.g., move file, change permissions, etc.
            } else {
                // Chunk uploaded successfully, more chunks to come
                $response = [
                    'status' => 200,
                    'message' => 'Chunk upload successful'
                ];
            }
             // Clean up the temporary chunk file after successful append
            FileHandler::cleanupChunk($tempChunkPath);

        } else {
            error_log("Failed to append chunk {$chunkIndex} for file {$fileName}");
            $response = ['status' => 500, 'message' => 'Failed to process chunk.'];
        }
    }

    header('Content-Type: application/json');
    echo json_encode($response);
    exit;
} else {
    // Handle other request methods or types if necessary
    header("HTTP/1.1 405 Method Not Allowed");
    echo json_encode(['status' => 405, 'message' => 'Method Not Allowed']);
}
?>

Considerations:

  • Error Handling and Retries: Implement robust error handling on both client and server. Consider adding retry logic for failed chunk uploads.
  • Security: Sanitize all user inputs, especially filenames, to prevent directory traversal attacks. Validate file types and sizes on the server.
  • Progress Indication: Use the progress event of XMLHttpRequest to provide real-time upload progress feedback to the user.
  • Concurrency: Ensure your server-side code handles concurrent uploads gracefully, possibly using file locking mechanisms as demonstrated.
  • Chunk Size Optimization: The optimal chunk size can depend on network conditions and server capabilities. Experiment to find a balance between performance and overhead.
  • Server Configuration: Ensure your PHP configuration (e.g., upload_max_filesize, post_max_size, max_execution_time) is sufficient for handling potentially large requests, even if they are chunked.

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.