Android upload and download file progress monitoring, large file upload (within 500M)

1. Ordinary files are uploaded and downloaded in the form of retrofit+rxjava;

1.1 dependence

    implementation 'com.squareup.retrofit2:retrofit:2.4.0'
    implementation 'com.squareup.retrofit2:adapter-rxjava:2.4.0'
    implementation 'com.squareup.retrofit2:converter-gson:2.4.0'
    implementation 'io.reactivex:rxandroid:1.2.1'

1.2 retrofit interface

    @GET("{filePath}")
    Observable<ResponseBody> getFile(@Path("filePath") String filePath,
                                       @Query("token") String token);


    @Multipart
    @POST(ApiConstants.FILE_UPLOAD_URL)
    Observable<ResponseBody> uploadFile(@Part("type") RequestBody type,
                                    @PartMap Map<String, RequestBody> file);

    @Multipart
    @POST(ApiConstants.FILE_UPLOAD_URL)
    Observable<ResponseBody> uploadFile2(@Part("type") RequestBody type,
                                            @Part MultipartBody.Part file);

Upload files using @ Multiparty and @ post methods. Parameters can be transferred using @ PartMap (multiple files can be uploaded) or @ Part's MultipartBody

1.3 transmission

Upload more than 1.3 files

Map<String, RequestBody> fileMap = new HashMap<>();
            RequestBody photoRequestBody = RequestBody.create(MediaType.parse("application/octet-stream"), file1);
            fileMap.put("file\";filename=\"" + file1.getName(), photoRequestBody);
RequestBody photoRequestBody2 = RequestBody.create(MediaType.parse("application/octet-stream"), file2);
            fileMap.put("file\";filename=\"" + file2.getName(), photoRequestBody2);

Use the Create method of RequestBody to create an object and put it into the map;

ApiClient.initService(INoticeDetailBiz.class,
                ApiConstants.FILE_SERVER, listener)
                .uploadFile(RequestBody.create(MediaType.parse("text/plain"),"nstest",
                            fileMap)
                .compose(RxSchedulerUtils.normalSchedulersTransformer())
                .subscribe(new Observer<ResponseBody>() {
                    
                });

The above code is the calling method of rxjava, so I won't say more;

1.3.2 upload a single file multipart Part

RequestBody requestBody1 = RequestBody.create(MediaType.parse("application/octet-stream"), file1);
MultipartBody.Part partFile = MultipartBody.Part.createFormData("file", file1.getName(), requestBody1);

Use multipartbody Part. The createformdata method creates the type required by the interface

ApiClient.initService(INoticeDetailBiz.class,
                ApiConstants.FILE_SERVER, listener)
                .uploadFile2(RequestBody.create(MediaType.parse("text/plain"),"nstest",
                            partFile)
                .compose(RxSchedulerUtils.normalSchedulersTransformer())
                .subscribe(new Observer<ResponseBody>() {
                    
                });

The same rxjava call

1.4. ApiClient class

private static Converter.Factory gsonConverterFactory = GsonConverterFactory.create(new Gson());
    private static CallAdapter.Factory rxJavaCallAdapterFactory = RxJavaCallAdapterFactory.create();
public static <T> T initService(Class<T> clazz, String baseUrl,
                                    ProgressListener listener) {
        OkHttpClient.Builder builder = new OkHttpClient().newBuilder();
        builder.connectTimeout(connectionTime, TimeUnit.SECONDS);
        builder.readTimeout(readTime, TimeUnit.SECONDS);
        builder.writeTimeout(writeTime, TimeUnit.SECONDS);
        builder.addNetworkInterceptor(new StethoInterceptor());
        builder.addInterceptor(new ErrorInterceptor());
        HttpLoggingInterceptor httpLoggingInterceptor;
        if (BuildConfig.DEBUG) {
            httpLoggingInterceptor = new HttpLoggingInterceptor();
        } else {
            httpLoggingInterceptor = new HttpLoggingInterceptor(
                    message -> logger.debug(message));
        }
        httpLoggingInterceptor.setLevel(HttpLoggingInterceptor.Level.BODY);
        builder.addInterceptor(httpLoggingInterceptor);
        if (null != listener) {
            if (listener.isDownload()) {
                builder.addInterceptor(new DownloadInterceptor(listener));
            } else {
                builder.addInterceptor(new UploadInterceptor(listener));
            }
        }
        Retrofit retrofit = new Retrofit.Builder()
                .client(builder.build())
                .baseUrl(baseUrl)
                .addConverterFactory(gsonConverterFactory)
                .addCallAdapterFactory(rxJavaCallAdapterFactory)
                .build();

        return retrofit.create(clazz);
    }

Here, three new objects are found

ProgressListener / / the interface for monitoring progress
DownloadInterceptor / / downloaded interceptor
UploadInterceptor / / uploaded interceptor

2. Implementation of upload and download progress monitoring

2.1 progress monitoring interface

public interface ProgressListener {
    void onProgress(int currentLength);//The current progress is already a percentage
    boolean isDownload();//Returning true means downloading and false means uploading
}

2.2 upload progress monitoring

Upload Interceptor

package com.dtrt.preventpro.myhttp.download;

import java.io.IOException;

import okhttp3.Interceptor;
import okhttp3.Request;
import okhttp3.Response;

public class UploadInterceptor implements Interceptor {
    private ProgressListener listener;

    public UploadInterceptor(ProgressListener listener) {
        this.listener = listener;
    }
    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request();
        Request build = request.newBuilder().post(new ProgressRequestBody(request.body(), listener)).build();
        return chain.proceed(build);
    }
}

ProgressRequestBody.java

package com.dtrt.preventpro.myhttp.download;

import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okhttp3.ResponseBody;
import okio.Buffer;
import okio.BufferedSink;
import okio.BufferedSource;
import okio.ForwardingSink;
import okio.ForwardingSource;
import okio.Okio;
import okio.Sink;
import okio.Source;

public class ProgressRequestBody extends RequestBody {
    private final RequestBody requestBody;
    private final ProgressListener progressListener;
    private BufferedSink bufferedSink;

    public ProgressRequestBody(RequestBody requestBody, ProgressListener progressListener) {
        this.requestBody = requestBody;
        this.progressListener = progressListener;
    }

    @Override
    public MediaType contentType() {
        return requestBody.contentType();
    }


    @Override
    public long contentLength() throws IOException {
        return requestBody.contentLength();
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException {
        if (bufferedSink == null) {
            //packing
            Sink sk = sink(sink);
            bufferedSink = Okio.buffer(sk );
        }
        //write in
        requestBody.writeTo(bufferedSink);
        //flush must be called, otherwise the last part of the data may not be written
        bufferedSink.flush();
    }

    private Sink sink(Sink sink) {
        return new ForwardingSink(sink) {
            //Current bytes written
            long bytesWritten = 0L;
            //Total byte length. Avoid calling contentLength() method multiple times
            long contentLength = 0L;

            @Override
            public void write(Buffer source, long byteCount) throws IOException {
                super.write(source, byteCount);
                if (contentLength == 0) {
                    //Obtain the value of contentLength and do not call it later
                    contentLength = contentLength();
                }
                //Increase the number of bytes currently written
                bytesWritten += byteCount;
                //Callback
                if (progressListener != null) {
                    progressListener.onProgress((int)(bytesWritten*1.0f/requestBody.contentLength()*100));
                }
            }
        };
    }

}

In fact, it inherits the RequestBody, passes in the progress listener, and operates the buffer in the writeTo() method; Among them, the number of bytes written will be accumulated, and then the percentage will be calculated, which will be called back through the progress interface; Therefore, a ProgressListener object is passed in the rxjava call in the above code.

private ProgressListener uploadListener = new ProgressListener() {
        @Override
        public void onProgress(int currentLength) {
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    tv_progress.setText("Uploading file..."+currentLength+"%");
                }
            });
        }
        @Override
        public boolean isDownload() {
            return false;
        }
    };

This is where the update progress of the page is written. After passing this object to the ApiClient method; It will automatically add the required interceptors; After the interceptor monitors the progress change, it will call back and update the ui; Note: the progress here is called back in the sub thread, and it needs to be sent to the main thread to update the ui.

2.3. Similarly, let's look at download progress monitoring

Download interceptor

package com.dtrt.preventpro.myhttp.download;

import java.io.IOException;

import okhttp3.Interceptor;
import okhttp3.Request;
import okhttp3.Response;

public class DownloadInterceptor implements Interceptor {
    private ProgressListener downloadListener;

    public DownloadInterceptor(ProgressListener downloadListener) {
        this.downloadListener = downloadListener;
    }

    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request();
        Response response = chain.proceed(request);
        return response.newBuilder()
                .body(new ProgressResponseBody(response.body(), downloadListener))
                .build();
    }
}

ProgressResponseBody.java

package com.dtrt.preventpro.myhttp.download;

import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.ResponseBody;
import okio.Buffer;
import okio.BufferedSource;
import okio.ForwardingSource;
import okio.Okio;
import okio.Source;

public class ProgressResponseBody extends ResponseBody {
    private final ResponseBody responseBody;
    private final ProgressListener progressListener;
    private BufferedSource bufferedSource;

    public ProgressResponseBody(ResponseBody responseBody, ProgressListener progressListener) {
        this.responseBody = responseBody;
        this.progressListener = progressListener;
    }

    @Override
    public MediaType contentType() {
        return responseBody.contentType();
    }


    @Override
    public long contentLength() {
        return responseBody.contentLength();
    }

    @Override
    public BufferedSource source() {
        if (bufferedSource == null) {
            bufferedSource = Okio.buffer(source(responseBody.source()));
        }
        return bufferedSource;
    }

    private Source source(Source source) {
        return new ForwardingSource(source) {
            long totalBytesRead = 0L;
            @Override
            public long read(Buffer sink, long byteCount) throws IOException {
                long bytesRead = super.read(sink,byteCount);
                totalBytesRead += bytesRead != -1 ? bytesRead : 0;
                if (progressListener != null) {
                    progressListener.onProgress((int)(totalBytesRead*1.0f/responseBody.contentLength()*100));
                }
                return bytesRead;
            }
        };
    }
}

Similarly, inherit the ResponseBody, pass in the progress listener, and operate the buffer in the source() method; Among them, the number of bytes read will be accumulated, and then the percentage will be calculated, which will be called back through the progress interface; Therefore, a ProgressListener object is passed in the rxjava call in the above code.

private ProgressListener downloadListener = new ProgressListener() {
        @Override
        public void onProgress(int currentLength) {
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    tv_progress.setText("Downloading files..."+currentLength+"%");
                }
            });
        }
        @Override
        public boolean isDownload() {
            return true;
        }
    };

This is where the update progress of the page is written. After passing this object to the ApiClient method; It will automatically add the required interceptors; After the interceptor monitors the progress change, it will call back and update the ui; Note: the progress here is called back in the sub thread, and it needs to be sent to the main thread to update the ui. The isDownload() method inside: returns true when downloading and false when uploading.

So far, the ordinary file upload, download and progress monitoring have been completed; However, our project should support the upload of large files below 200M; Through the above code test, it is found that Huawei mobile phone can upload 70M files, while vivo mobile phone can only upload 50M files; Different mobile phones can support different upload sizes; But those over 10M belong to large files. Here is a way to upload large files.

3. Large file upload

If you find that you can upload using apache's httpclient on the Internet, let's try it;

3.1 adding dependencies

//httpclient upload large file function
    implementation group: 'org.apache.httpcomponents.client5' , name: 'httpclient5' , version: '5.0'
    implementation 'org.slf4j:slf4j-android:1.7.22'

httpclient5 is used here.

3.2 upload

private void upload(File file, ProgressListener listener){
        HttpClient client = HttpClientBuilder.create().build();
        HttpPost httpPost = new HttpPost(ApiConstants.FILE_SERVER+ApiConstants.FILE_UPLOAD_URL);
        try {
            MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
            String filesKey = "file";
            multipartEntityBuilder.addPart(filesKey, new MyFileBody(file, new WriteListener() {
                @Override
                public void registerWrite(long amountOfBytesWritten) {
                    System.out.println("===========>"+amountOfBytesWritten);
                    listener.onProgress((int)(amountOfBytesWritten * 1.0f / notice.getFile().length() * 100));
                }
            }));
            // The second file (if there are multiple files, just use the same key, and the back end can receive them with an array or set)
            //File file2 = new File("C:\Users\JustryDeng\Desktop \ \ avatar. jpg");
            // Prevent the file name received by the server from being garbled. Here, we can first the file name URLEncode, and then the server gets the file name in URLEncode. Can avoid the problem of garbled code.
            // The file name is actually transferred in the content disposition of the request header, such as form data; name="files";  Filename = "Avatar. jpg"
            //multipartEntityBuilder.addBinaryBody(filesKey, file2, ContentType.DEFAULT_BINARY, URLEncoder.encode(file2.getName(), "utf-8"));
            // Other parameters (Note: customize contentType and set UTF-8 to prevent the parameters obtained by the server from being garbled)
            ContentType contentType = ContentType.create("text/plain", Charset.forName("UTF-8"));
            multipartEntityBuilder.addTextBody("type", "nstest", contentType);
            HttpEntity httpEntity = multipartEntityBuilder.build();
            httpPost.setEntity(httpEntity);
            //Other parameters such as timeout can be configured here
            //RequestConfig requestConfig = RequestConfig
            //   .custom()
            //   .setConnectionRequestTimeout(10, TimeUnit.MINUTES)
            //   .setConnectTimeout(10, TimeUnit.MINUTES)
            //   .setResponseTimeout(10, TimeUnit.MINUTES)
            //   .build();
            //httpPost.setConfig(requestConfig);

            CloseableHttpResponse response = (CloseableHttpResponse) client.execute(httpPost);
            System.out.println("HTTPS Response status is:" + response.getCode());
            if (response.getCode() == 200 && response.getEntity() != null) {
                System.out.println("HTTPS Response content length is:" + response.getEntity().getContentLength());
                // Actively set the code to prevent the response from being garbled
                String responseStr = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
                System.out.println("HTTPS The response content is:" + responseStr);
                if (!TextUtils.isEmpty(responseStr)) {
                    notice.setCodes(responseStr);
                    commit(notice);
                }else {
                    //Upload failed
                }
            }else {
                //Upload failed
            }
        } catch (Exception e) {
            e.printStackTrace();
            //Upload failed
        }
    }

In fact, it is similar to the uploaded file above; Is to package the file as an object in httpclient; Then judge whether the upload is successful according to the return code. Here is an article written in detail: https://blog.csdn.net/justry_deng/article/details/81042379

Focus here:

multipartEntityBuilder.addPart(filesKey, new MyFileBody(notice.getFile(), new WriteListener() {
    @Override
    public void registerWrite(long amountOfBytesWritten) {
        System.out.println("===========>"+amountOfBytesWritten);
        listener.onProgress((int)(amountOfBytesWritten * 1.0f / notice.getFile().length() * 100));
    }
}));

When adding files to the Entity object, MyFileBody is used:

package com.dtrt.preventpro.myhttp.download;

import org.apache.hc.client5.http.entity.mime.FileBody;

import java.io.File;
import java.io.IOException;
import java.io.OutputStream;

public class MyFileBody extends FileBody {
    private WriteListener listener;
    public MyFileBody(File file, WriteListener listener) {
        super(file);
        this.listener = listener;
    }

    @Override
    public void writeTo(OutputStream out) throws IOException {
        OutputStreamProgress output = new OutputStreamProgress(out, listener);
        super.writeTo(output);
    }
    public interface WriteListener {
        void registerWrite(long amountOfBytesWritten);
    }
    public class OutputStreamProgress extends OutputStream {
        private final OutputStream outstream;
        private long bytesWritten=0;
        private final WriteListener writeListener;
        public OutputStreamProgress(OutputStream outstream, WriteListener writeListener) {
            this.outstream = outstream;
            this.writeListener = writeListener;
        }
        @Override
        public void write(int b) throws IOException {
            outstream.write(b);
            bytesWritten++;
            writeListener.registerWrite(bytesWritten);
        }
        @Override
        public void write(byte[] b) throws IOException {
            outstream.write(b);
            bytesWritten += b.length;
            writeListener.registerWrite(bytesWritten);
        }
        @Override
        public void write(byte[] b, int off, int len) throws IOException {
            outstream.write(b, off, len);
            bytesWritten += len;
            writeListener.registerWrite(bytesWritten);
        }
        @Override
        public void flush() throws IOException {
            outstream.flush();
        }
        @Override
        public void close() throws IOException {
            outstream.close();
        }
    }
}

Principle: MyFileBody inherits FileBody. When the source code accepts the output stream in the writeTo() method, we replace it with the output stream with a listener; At this time, when the source code calls the write operation, the listener can be triggered to monitor the progress of the write.

OutputStreamProgress inherits from the output stream OutputStream, passes in the listener, rewrites each write method, accumulates the written bytes during the write operation, and triggers the listener; In this way, the upload of large files is realized.

The progress update of the page is the same as above. Using this method of transmission, my test result is that files below 500M can be uploaded, and those greater than 500M will report an error of Socket pipe fracture. But it's enough. After all, it's a mobile phone, and there is little place for large file transmission.

 

 

 

 

Tags: Android

Posted by alan007 on Sun, 15 May 2022 23:45:24 +0300