2

Cloud Storage FUSE

 1 year ago
source link: https://cloud.google.com/storage/docs/gcs-fuse
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Cloud Storage FUSE

bookmark_border

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA products and features might have limited support, and changes to pre-GA products and features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

This page provides an overview of Cloud Storage FUSE, an open source FUSE adapter that allows you to mount and access Cloud Storage buckets as local file systems, so applications can read and write objects in your bucket using standard file system semantics.

See the Cloud Storage FUSE GitHub documentation for instructions on how to use Cloud Storage FUSE, such as installing the gcsfuse CLI and mounting buckets.

How Cloud Storage FUSE works

Cloud Storage FUSE uses FUSE and Cloud Storage APIs to transparently expose buckets as locally mounted folders on your file system. Cloud Storage FUSE can be run from anywhere with connectivity to Cloud Storage, including Google Kubernetes Engine, Compute Engine VMs, or on-premises systems.

Cloud Storage FUSE works by translating object storage names into a file and directory system, interpreting the slash character ("/") in object names as a directory separator so that objects with the same common prefix are treated as files in the same directory. Applications can interact with the mounted bucket like a file system, providing virtually limitless file storage running in the cloud.

Cloud Storage FUSE is ideal for use cases where Cloud Storage has the right performance and scalability characteristics for an application that requires file system semantics. For example, Cloud Storage FUSE is useful for machine learning (ML) projects because it provides a way to store data, models, checkpoints, and logs directly in Cloud Storage. For more information, see Cloud Storage FUSE for ML workloads.

Differences from other file systems

While Cloud Storage FUSE has a file system interface, it is not like an NFS or CIFS file system on the backend. Additionally, Cloud Storage FUSE is not POSIX compliant. For a POSIX file system product in Google Cloud, see Filestore.

Supported applications and platforms

Cloud Storage FUSE officially supports the TensorFlow V2.x, TensorFlow V1.x, and PyTorch V1.x ML frameworks. Support for usage with other frameworks is not guaranteed.

Cloud Storage FUSE supports the following operating systems running on Linux kernel versions 3.10 and earlier:

  • Ubuntu 18.04 or later

  • Debian 10 or later

  • CentOS 7 or later

  • RHEL 7 or later

Get support

You can get support, submit general questions, and request new features by filing issues in GitHub. You can also get support by using one of Cloud Storage's official support channels.

For solutions to commonly-encountered issues, see Troubleshooting in the Cloud Storage FUSE GitHub documentation.

Pricing for Cloud Storage FUSE

Cloud Storage FUSE is available free of charge, but the storage, metadata, and network I/O it generates to and from Cloud Storage are charged like any other Cloud Storage interface. In other words, all data transfer and operations performed by Cloud Storage FUSE map to Cloud Storage transfers and operations, and are charged accordingly. For more information on common Cloud Storage FUSE operations and how they map to Cloud Storage operations, see the operations mapping.

To avoid surprises, you should estimate how your use of Cloud Storage FUSE will translate to Cloud Storage charges. For example, if you are using Cloud Storage FUSE to store log files, you can incur charges quickly if logs are aggressively flushed on hundreds or thousands of machines at the same time.

Be aware of the following categories of charges related to using Cloud Storage FUSE:

  • Normal object operations (create, delete, and list) incur charges as described in the Operations section of the Cloud Storage pricing page.

  • Nearline storage, Coldline storage, and Archive storage objects have costs associated with retrieval and early deletion.

  • Network egress and data transfer between locations incur costs. For more information, see the Network section in the Cloud Storage pricing page.

Map of Cloud Storage FUSE operations to Cloud Storage operations

When you perform an operation using Cloud Storage FUSE, you also perform the Cloud Storage operations associated with the Cloud Storage FUSE operation. The following table describes common Cloud Storage FUSE commands and their associated Cloud Storage JSON API operations. You can display information about the Cloud Storage FUSE operations by using the --debug_gcs flag.

Command JSON API Operations
gcsfuse --debug_gcs example-bucket mp Objects.list (to check credentials)
cd mp n/a
ls mp Objects.list("")
mkdir subdir

Objects.get("subdir")

Objects.get("subdir/")

Objects.insert("subdir/")
cp ~/local.txt subdir/

Objects.get("subdir/local.txt")

Objects.get("subdir/local.txt/")

Objects.insert("subdir/local.txt"), to create an empty object

Objects.insert("subdir/local.txt"), when closing after done writing
rm -rf subdir

Objects.list("subdir")

Objects.list("subdir/")

Objects.delete("subdir/local.txt")

Objects.list("subdir/")

Objects.delete("subdir/")

Next steps


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK