3
Amazon S3 对象存储Java API操作记录（Minio与S3 SDK两种实现） - 东北小狐狸
source link: https://www.cnblogs.com/hellxz/p/17359828.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求，由于4月份自学考试临近，一直在备考就拖着没总结记录下，开发联调过程中也出现过一些奇葩的问题，最近人刚从考试缓过来顺手记录一下。
S3对象存储的基本概念
S3是什么？
Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API，于是成为了对象存储服务(Object Storage Service，OSS)业内的标准接口规范。
S3的逻辑模型
如下图，我们可以把S3的存储空间想象成无限的，想存储一个任意格式的文件到S3服务中，只需要知道要把它放到哪个桶(Bucket)中，它的名字（Object Id）应该是什么。
按图中的模型，可简单理解为S3是由若干个桶（Bucket）组成，每个桶中包含若干个不同标识的对象（Object），还有就是统一的访问入口(RESTful API)，这样基本就足够了。
Minio客户端方式操作S3
详细API文档：https://min.io/docs/minio/linux/developers/java/API.html
以下代码异常处理做了简化，真实使用时请注意捕获异常做处理。
Maven:
<dependency> <groupId>io.minio</groupId> <artifactId>minio</artifactId> <version>8.5.2</version></dependency>
Gradle:
dependencies { implementation("io.minio:minio:8.5.2")}
初始化客户端
private static final String HTTP_PROTOCOL = "http"; private MinioClient minioClient;private String endpoint = "http://192.168.0.8:9200";private String accessKey = "testKey";private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); try { // url上无端口号时，识别http为80端口，https为443端口 int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort(); boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true; //@formatter:off this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security) .credentials(accessKey, secretKey).build(); //@formatter:on // 忽略证书校验，防止自签名证书校验失败导致无法建立连接 this.minioClient.ignoreCertCheck(); } catch (Exception e) { e.printStackTrace(); }}
public boolean createBucket(String bucket) { try { minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; } return true;}
public boolean deleteBucket(String bucket) { try { minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build()); logger.info("删除桶[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true;}
判断桶是否存在
public boolean bucketExists(String bucket) { try { return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; }}
public void upload(String bucket, String objectId, InputStream input) { try { //@formatter:off minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId) .stream(input, input.available(), -1) .build()); //@formatter:on } catch (Exception e) { e.printStackTrace(); }}
提供两个下载方法，一个将输入流返回，另一个用参数输出流写出
public InputStream download(String bucket, String objectId) { try { return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } return null;} public void download(String bucket, String objectId, OutputStream output) { //@formatter:off try (InputStream input = minioClient.getObject( GetObjectArgs.builder().bucket(bucket).object(objectId).build())) { IOUtils.copyLarge(input, output); } catch (Exception e) { e.printStackTrace(); } //@formatter:on}
public boolean deleteObject(String bucket, String objectId) { //@formatter:off try { minioClient.removeObject(RemoveObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } //@formatter:on return true;}
判断对象是否存在
public boolean objectExists(String bucket, String key) { //@formatter:off try { // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在 minioClient.statObject(StatObjectArgs.builder() .bucket(bucket).object(key).build()); } catch (Exception e) { return false; } //@formatter:on return true;}
import java.io.InputStream;import java.io.OutputStream;import java.net.MalformedURLException;import java.net.URL; import org.apache.tomcat.util.http.fileupload.IOUtils;import org.slf4j.Logger;import org.slf4j.LoggerFactory; import io.minio.BucketExistsArgs;import io.minio.GetObjectArgs;import io.minio.MakeBucketArgs;import io.minio.MinioClient;import io.minio.PutObjectArgs;import io.minio.RemoveBucketArgs;import io.minio.RemoveObjectArgs;import io.minio.StatObjectArgs; public class S3MinioClientDemo { private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class); private static final String HTTP_PROTOCOL = "http"; private MinioClient minioClient; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); try { // url上无端口号时，识别http为80端口，https为443端口 int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort(); boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true; //@formatter:off this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security) .credentials(accessKey, secretKey).build(); //@formatter:on // 忽略证书校验，防止自签名证书校验失败导致无法建立连接 this.minioClient.ignoreCertCheck(); } catch (Exception e) { e.printStackTrace(); } } public boolean createBucket(String bucket) { try { boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); if (found) { logger.info("桶名[{}]已存在", bucket); return false; } minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); } return true; } public boolean deleteBucket(String bucket) { try { minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build()); logger.info("删除桶[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public boolean bucketExists(String bucket) { try { return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; } } public void upload(String bucket, String objectId, InputStream input) { try { //@formatter:off minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId) .stream(input, input.available(), -1) .build()); //@formatter:on } catch (Exception e) { e.printStackTrace(); } } public InputStream download(String bucket, String objectId) { try { return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream output) { //@formatter:off try (InputStream input = minioClient.getObject( GetObjectArgs.builder().bucket(bucket).object(objectId).build())) { IOUtils.copyLarge(input, output); } catch (Exception e) { e.printStackTrace(); } //@formatter:on } public boolean objectExists(String bucket, String objectId) { //@formatter:off try { // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在 minioClient.statObject(StatObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { return false; } //@formatter:on return true; } public boolean deleteObject(String bucket, String objectId) { //@formatter:off try { minioClient.removeObject(RemoveObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } //@formatter:on return true; } public void close() { minioClient = null; } }
Amazon S3 SDK方式操作S3
官方API文档：https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
这里由于项目上提供的SDK和文档都是1.x的，这里就暂时只提供1.x的代码
Maven:
<dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk-s3</artifactId> <version>1.11.300</version></dependency>
Gradle:
dependencies { implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'}
初始化客户端
private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class); private AmazonS3 s3client;private String endpoint = "http://192.168.0.8:9200";private String accessKey = "testKey";private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); String protocol = endpointUrl.getProtocol(); int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort(); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setSignerOverride("S3SignerType"); clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase())); // 禁用证书检查，避免https自签证书校验失败 System.setProperty("com.amazonaws.sdk.disableCertChecking", "true"); // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题 System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true"); AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey); // 创建 S3Client 实例 AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig); s3client.setEndpoint(endpointUrl.getHost() + ":" + port); s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build()); this.s3client = s3client;}
public boolean createBucket(String bucket) { String bucketName = parseBucketName(bucket); try { if (s3client.doesBucketExist(bucketName)) { logger.warn("bucket[{}]已存在", bucketName); return false; } s3client.createBucket(bucketName); } catch (Exception e) { e.printStackTrace(); } return true;}
public boolean deleteBucket(String bucket) { try { s3client.deleteBucket(bucket); logger.info("删除bucket[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true;}
判断桶是否存在
public boolean bucketExists(String bucket) { try { return s3client.doesBucketExist(bucket); } catch (Exception e) { e.printStackTrace(); } return false;}
public void upload(String bucket, String objectId, InputStream input) { try { // 创建文件上传的元数据 ObjectMetadata meta = new ObjectMetadata(); // 设置文件上传长度 meta.setContentLength(input.available()); // 上传 s3client.putObject(bucket, objectId, input, meta); } catch (Exception e) { e.printStackTrace(); }}
public InputStream download(String bucket, String objectId) { try { S3Object o = s3client.getObject(bucket, objectId); return o.getObjectContent(); } catch (Exception e) { e.printStackTrace(); } return null;} public void download(String bucket, String objectId, OutputStream out) { S3Object o = s3client.getObject(bucket, objectId); try (InputStream in = o.getObjectContent()) { IOUtils.copyLarge(in, out); } catch (Exception e) { e.printStackTrace(); }}
public boolean deleteObject(String bucket, String objectId) { try { s3client.deleteObject(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } return true;}
判断对象是否存在
public boolean existObject(String bucket, String objectId) { try { return s3client.doesObjectExist(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; }}
import java.io.InputStream;import java.io.OutputStream;import java.net.MalformedURLException;import java.net.URL; import org.apache.tomcat.util.http.fileupload.IOUtils;import org.slf4j.Logger;import org.slf4j.LoggerFactory; import com.amazonaws.ClientConfiguration;import com.amazonaws.Protocol;import com.amazonaws.auth.AWSCredentials;import com.amazonaws.auth.BasicAWSCredentials;import com.amazonaws.services.s3.AmazonS3;import com.amazonaws.services.s3.AmazonS3Client;import com.amazonaws.services.s3.S3ClientOptions;import com.amazonaws.services.s3.model.ObjectMetadata;import com.amazonaws.services.s3.model.S3Object; /** * S3对象存储官方SDK实现 * * @author ZhangChenguang * @date 2023年2月2日 */@SuppressWarnings("deprecation")public class S3SdkDemo { private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class); private AmazonS3 s3client; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); String protocol = endpointUrl.getProtocol(); int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort(); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setSignerOverride("S3SignerType"); clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase())); // 禁用证书检查，避免https自签证书校验失败 System.setProperty("com.amazonaws.sdk.disableCertChecking", "true"); // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题 System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true"); AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey); // 创建 S3Client 实例 AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig); s3client.setEndpoint(endpointUrl.getHost() + ":" + port); s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build()); this.s3client = s3client; } public boolean createBucket(String bucket) { try { s3client.createBucket(bucket); } catch (Exception e) { e.printStackTrace(); } return true; } public boolean deleteBucket(String bucket) { try { s3client.deleteBucket(bucket); logger.info("删除bucket[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public boolean bucketExists(String bucket) { try { return s3client.doesBucketExist(bucket); } catch (Exception e) { e.printStackTrace(); } return false; } public void upload(String bucket, String objectId, InputStream input) { try { // 创建文件上传的元数据 ObjectMetadata meta = new ObjectMetadata(); // 设置文件上传长度 meta.setContentLength(input.available()); // 上传 s3client.putObject(bucket, objectId, input, meta); } catch (Exception e) { e.printStackTrace(); } } public InputStream download(String bucket, String objectId) { try { S3Object o = s3client.getObject(bucket, objectId); return o.getObjectContent(); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream out) { S3Object o = s3client.getObject(bucket, objectId); try (InputStream in = o.getObjectContent()) { IOUtils.copyLarge(in, out); } catch (Exception e) { e.printStackTrace(); } } public boolean existObject(String bucket, String objectId) { try { return s3client.doesObjectExist(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } } public boolean deleteObject(String bucket, String objectId) { try { s3client.deleteObject(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public void close() { s3client = null; }}
遇到的问题
1、bucket名称必须是小写，不支持下划线
- 处理方式：写方法转换下bucket名称，将大写转小写，将下划线替换为中划线。
2、minio客户端下载非官方S3存储的文件时，如果响应头的Content-Length与实际文件大小不符，会导致minio客户端包装的okhttp3报错
报错信息：
Caused by: java.net.ProtocolException: unexpected end of stream at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?] at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?] at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?] at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?] at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?] ... 122 more
抓包发现问题的图：
最终换成了S3官方SDK可用了。
PS：客户现场部署的S3是浪潮公司提供的，如果现场遇到这个情况，就不要固执去找对方对线了，完全没用。。
S3存储的基本操作就记录到这里了，由于没有S3存储就没尝试官方SDK的V2版本，由于这些代码是总结时从业务代码里抽取出来的，可能会有点问题，但大体思路已经有了。
希望对读者有所用处，觉得写得不错和有帮到你，欢迎点个赞，您的支持就是我的鼓励！
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK