Skip to content

8.9 gfs Object Storage

1 Quick Start

1.1 Login

Visithttps://gfs.ihep.ac.cn/and log in with your IHEP unified authentication account.

Obtain the Access Key and Secret Key from the web interface.

1.2 Linux Client Mounting

  • Install thejcliclient.

    Select thejclidownload link for your CPU architecture:

    For example, on Linux, usewgetorcurlto download thejcliclient package.

    wgethttps://file-ocloud.ihep.ac.cn/jcli/amd64/jclicurl-Lhttps://file-ocloud.ihep.ac.cn/jcli/amd64/jcli>jcli# Grant execute permissionchmod+xjcli

    Configure it under the user'sHOME/bindirectory so that thejclicommand can be used directly from the command line.

    mkdir$HOME/bin/cpjcli$HOME/bin/exportPATH=$PATH:$HOME/binecho"export PATH=$PATH:$HOME/bin">>~/.bashrc
  • Configure authentication information.

    jcliauth
    • Enter the Access Key and Secret Key.
    • Enter the gateway address (endpoint):https://fgws3-gfs.ihep.ac.cn
    • Proxy can be left blank.
    • The default bucket can be selected as0.

    Example:

    jcliauthEnterAccessKey(currentvalue:[]):WKYxxxxxxxxxxxxxxxb4EnterSecretKey(currentvalue:[]):ZEDxxxxxxxxxxxxxxxpZEnterendpoints,separatedbycommas(currentvalue:[]):https://fgws3-gfs.ihep.ac.cnEnterproxyinformation(optional):SelectdefaultBucket:>xxxxxx-testbucket01[list,read,write]Configurationfilesavedsuccessfully:./.jcli/token/auth
  • View bucket details.

    jclils

    If the command outputs information without errors, the configuration is correct and subsequent operations can be performed.

  • Mount the bucket.

    jclimount-f{mount_point}--daemon

    Example:

    housiqi@lxlogin003:~$mkdir/scratchfs2/cc/housiqi/gfs_bucket# Create a mount pointhousiqi@lxlogin003:~$jclimount-f/scratchfs2/cc/housiqi/gfs_bucket/--daemon# MountHint:Startingchildprocesstoruntheservice...housiqi@lxlogin003:~$ls/scratchfs2/cc/housiqi/gfs_bucket/# Verify the mount'Jwanfs User Manual.pdf'mydirtmpdemo-directoryhousiqi@lxlogin003:~$df-h# View mount status20005-ihep-imel-120624512G3.0M512G1%/scratchfs2/cc/housiqi/gfs_bucket

    You can then access files in the bucket as if they were in a local directory.

1.3 Windows Client Mounting

  • Download links:

  • Configure environment variables and add the directory containingjcli.exetoPath.

    1. Locate thejclifile.
    2. Right-click the Windows icon in the lower-left corner and selectSystem.
    3. ClickAdvanced system settingson the left.
    4. InSystem Properties, clickEnvironment Variables.
    5. UnderSystem variables, findPathand clickEdit.
    6. ClickNewand enter the directory path wherejcliis located.
    7. ClickOKto save the change.
  • Download and install WinFSP:download link.

  • Configure authentication information.

    PressWin + R, entercmd, and runjcli authin the command prompt to configure AK, SK, and endpoint information as on Linux. After configuration, run the following command to mount the bucket:

    jclimount-fZ:--daemon

2 Web Access

Web address:https://gfs.ihep.ac.cn/

Apply for permission and log in with your IHEP unified authentication account. After the account is enabled, 1 TB of default storage space is provided automatically.

2.1 Bucket Management

The system provides functions such as creation, deletion, capacity expansion, and recycle bin management.

2.2 File Management

Click the bucket name to enter the file management interface.

Files in a bucket follow a network-disk-style user operation and data management model. The following functions are provided:

  • Publish resources
  • Upload files
  • Download files
  • Create directories
  • Preview files. On the file list page, click...and choose preview from the menu. The file can be previewed without downloading it locally.
  • Delete files. Files in the recycle bin are kept for30 daysby default.
  • Share files. Sharing duration and password can be configured.

File management function list

2.2.1 Upload and Download Files

On the file list page, click upload to start the upload process.

More files can be added or upload can be confirmed in the upload dialog.

Right-click a file and click download to download it.

2.2.2 File Sharing

Click file sharing to generate a shared link and access password.

Shared files can be viewed in the sharing list, where sharing information can also be copied.

Shared file list

Note

  • The sharing period and access password can be customized.
  • Only files can be shared; directories cannot be shared.
  • If the shared target is deleted, the sharing link remains valid, but the file cannot be downloaded.

2.2.3 File Preview

On the file list page, click...and choose preview from the menu to preview a file directly.

Preview availability depends on the file type. Supported file types include:

  • Images:jpg,png, etc.
  • Videos:mp4,avi, etc.
  • Documents:pdf,docx, etc.
  • Audio:mp3,wav, etc.
  • Text:txt,md, etc.
  • High-energy physics data files:root,h5

2.3 Resource Publishing

Users can publishfiles/directoriesto the resource square for other platform users to download.

Before publishing resources for the first time, create a shop on theMy Shoppage by filling in its name, overview, category, and tags.

Add a shop

On the file list page, click...and choose publish resource from the menu to start the publishing process.

Publish resource

Enter the resource information to publish the corresponding file or directory to the resource square for download.

On theMy Shoppage, you can view published resources and obtain download links. The shop and resources can also be viewed in the resource square.

2.4 Key Management (Important)

Keys (AK and SK) are theonly credentialsused by the system to identify users. Operations that access the gateway with a key represent actions by the key owner. Keep your authentication information secure.

2.5 Resource Square

Resources published by users can be viewed in detail and downloaded from here.

To quickly find a specific resource, use the search function. Enter resource keywords in the search box on the resource square page.

Resource square

2.6 Personal Settings

You can modify your nickname, email, and other information.

3 Official Client

JWanFS provides the officialjcliclient. Direct access through the official client is recommended. JWanFS also supports the standard S3 interface, so S3 clients can access platform resources directly.

3.1 Software Download

Download links:

CPU Architecture / OS Windows macOS Linux
Intel/AMD 64-bit (amd64/x86) Download Download Download
ARM/Apple Silicon (arm64/aarch64) Download Download Download

3.2 Configure Environment Variables

  • Linux

    • One-click installation:

      curlhttps://file-ocloud.ihep.ac.cn/get-jcli.sh|bash
    • Manual installation (recommended):

      Download the client program.

      wget{download_link}

      Copyjclito the user'sHOME/bindirectory.

      mkdir$HOME/bin/cpjcli$HOME/bin/exportPATH=$PATH:$HOME/binecho"export PATH=$PATH:$HOME/bin">>~/.bashrc

      Or configure it under/usr/binas root.

      sudocpjcli/usr/bin/

      After installation, runjcli versionto confirm that installation succeeded.

  • Windows

    On Windows, download the client executable from the download link and addjclito the environment variables for global use:

    1. Locate thejclifile.
    2. Right-click the Windows icon in the lower-left corner and selectSystem.
    3. ClickAdvanced system settingson the left.
    4. InSystem Properties, clickEnvironment Variables.
    5. UnderSystem variables, findPathand clickEdit.
    6. ClickNewand enter the directory path wherejcliis located.
    7. ClickOKto save the change.

3.3 Fill in Authentication Information

3.3.1 Manual Configuration

Copy the AK, SK, and file gateway obtained from the web interface, configure the client, and select the corresponding bucket.

Key information from the web interface:

Gateway information from the web interface:

You can also clickAccess Guideon the key management page to view available gateway nodes.

Command for configuring authentication information:

jcliauth# or reset configuration with:jcliauth-e

Parameters:

  • AccessKey/SecretKey: Enter the keys obtained from the web interface.
  • Endpoint: Enter the gateway selected from the web interface, for examplehttps://fgws3-gfs.ihep.ac.cn.
  • Bucket selection: Follow the prompts to select the default bucket.

Note

jcli authgenerates the authentication file for the AK/SK pair. Laterjclioperations access gfs storage through this authentication file.

You can also usejcli auth -eto reset the configuration and re-enter authentication information.

3.3.2 Generate JCLI Token Configuration

ClickGenerate JCLI Tokenon the key management page.

Generate JCLI token

In the expanded tab, select the bucket and gateway in order, copy the generated token or command, and paste it into a console wherejclihas been installed to quickly completejcliauthentication configuration.

JCLI token configuration

3.3.3 Configuration File Description

jclisupports specifying the authentication file location. The default location is~/.jcli/token/auth.

jcli-auth=/your/path/auth-file-nameauth

After the authentication file is created, add-auth=/your/path/auth-file-namewhen operating with this file.

Example:

jcli-auth=/your/path/auth-file-namels/

To set it as the default authentication file, add-default:

jcli-auth=/your/path/auth-file-nameauth-default

3.4 Mount as a Virtual Disk

  • Linux

    • Ensure FUSE is installed. It is usually built in. If not installed, use one of the following commands.

      For RPM-based systems such as CentOS/openEuler:

      sudoyuminstallfusefuse3

      For DEB-based systems such as Ubuntu/Debian:

      sudoaptinstallfuse
    • Ensurejcli authhas been configured.

      Runjcli ls. If it correctly lists files without errors, the configuration is correct.

    • Mount as a shared directory.

      Configure permissions after mounting:

      jclimount-f{mount_point}--daemon

      Example 1: Change/tmp/hostfile permission to0644(rw-r--r--).

      # Implicit parametersjclichmod0644/tmp/host# Explicit parametersjclichmod-r/tmp/host-mode0644

      Example 2: Change the owner of/tmp/hostto root.

      # Implicit parametersjclichownroot/tmp/host# Explicit parametersjclichown-r/tmp/host-uroot

      Specify access permissions when mounting:

      jclimount/lact\-bucket20001-lact\-mode-mnt"0770"\-mode-mnt-uid0\-mode-mnt-gid1000\-mode-uid0\-mode-gid1000\--daemon

      Parameter description:

      • -mode-mnt "0770": Only the owner (root) and group (lact) can read and write.
      • -mode-*-gid 1000: Force files to belong to the group with GID1000.
  • Windows

    • Download WinFSP:download link.
    • Ensurejcli authhas been configured.
    • Run the mount commandjcli mount {drive_letter}. The Windows drive letter must be selected from uppercaseA-Z.

      jclimountZ:
    • You can then use the JWanFS unified storage space like a local disk.

  • macOS

    Mounting on macOS requires MacFUSE.Download and install MacFUSE.

  • Mount path examples

Ifjcli mount Z:mounts the bucket toThis PC, orjcli mount /home/lhaaso/testuser/gfsmounts it to a Linux server, the path mapping is:

Path in Bucket Windows Linux
Image.jpg Z:\Image.jpg /home/lhaaso/testuser/gfs/Image.jpg

3.5 Unmount

Manual unmounting forces directory contents to be written back to disk and helps ensure data safety.

# Personal directoryumount/home/lhaaso/testuser/gfs# Shared directorysudoumount/lact

3.6 Interactive Mode

Runjclidirectly to enter interactive mode. It supports common commands such asls,cd,put, andget, and the usage is similar to a Linux shell.

  • Subcommands such asjcli putperform one-time file upload.
  • jcli mountprovides persistent mounting for long-term reading and writing.
  • jcli helpdisplays detailed command descriptions.
Global parameters:--help Value type: boolPrint help information-ak Value type: stringAccessKey-auth Value type: stringAuthentication file path-bucket Value type: stringDefault bucket-endpoint Value type: stringEndpoint address-sk Value type: stringSecretKeySubcommands:Command Alias Purposeauth Authentication servicecat View file contentschmod Modify object permissionschown Modify object permissionsconfig Configure authentication informationcp copy Copy objectcron crontab,task Scheduled task managementrm delete,del Delete filedu size Calculate file or directory sizeetag Calculate file ETagexpire Get AK/SK expiration timefile-server Start file serverftp Start FTP server for LAN usersget pull,download Download filehead View file headlist ls,ll List filesmd5sum md5 Calculate file MD5mkdir Create directorymount Mount bucketmv move,rename Move or rename remote fileput push,upload Upload filequota Get file informationstat stats,info Get file informationsymlink Modify object permissionssync Synchronize objectstail View file tailtoken Create temporary AK/SKupdate Update client version

3.7 Client Upgrade

To use new features, update thejcliclient with:

jcliupdate-d

4 Third-Party Compatible Clients

4.1 NetMount

NetMount is a simple, easy-to-use, and efficient open-source cloud storage management tool. It integrates cloud storage as a local file system, allowing seamless access to cloud resources. Through virtual file system technology, it enables efficient operations, simplifies data management, and provides a smooth cloud-local interaction experience.

Open NetMount, clickManage, clickAdd, and select object storage to continue.

Fill in bucket information to add the JWanFS bucket.

Storage Nameis the name shown in NetMount and can be customized.

Bucketis the JWanFS bucket.Endpointis the S3 gateway address of the file gateway.

Fill Access Key ID and Secret Access with the OSCA authentication AK and SK.

Note

Enableforce_path_style, selectList Object VersionasV2, and then save.

4.2 Cloud Volume

Cloud Volume is a Flutter desktop client for macOS, Windows, and Linux. It can manage S3-compatible object storage, WebDAV, and Baidu Netdisk in one place. It is more than a simple "bucket list + upload/download" tool: it includes local caching, mountable WebDAV views, an application-level recycle bin, sharing-link management, and task queues, making its workflow closer to a desktop file manager.

Cloud Volume project repository and release packages:

Project page:https://github.com/lfhy/cloud-volume

GitHub Release (macOS / Windows / Linux desktop versions + CLI): download the latest version from the repository Releases page.

5 SDK

gfs object storage is compatible with the S3 protocol and can be accessed through AWS S3 SDKs in multiple languages. Program access usually requires four configuration items:

  • Endpoint: gfs S3 gateway address, for examplehttps://fgws3-gfs.ihep.ac.cn
  • AccessKeyId/AccessKeySecret: access credentials
  • Region:us-east-1can be used
  • Path-Style: path-style access must be enabled to prevent the SDK from resolving the bucket name as a subdomain

5.1 Install SDK

gogetgithub.com/aws/aws-sdk-go
pip3installboto3

Add the following topom.xmlin a Maven project:

software.amazon.awssdks32.24.0
npminstall@aws-sdk/client-s3
composerrequireaws/aws-sdk-php

5.2 Initialize Client

ReplaceAccessKeyId,AccessKeySecret, andyour-bucket-namein the examples with actual information.

import("github.com/aws/aws-sdk-go/aws""github.com/aws/aws-sdk-go/aws/credentials""github.com/aws/aws-sdk-go/aws/session""github.com/aws/aws-sdk-go/service/s3")funcnewS3Client()*s3.S3{conf:=&aws.Config{Region:aws.String("us-east-1"),Endpoint:aws.String("https://fgws3-gfs.ihep.ac.cn"),Credentials:credentials.NewStaticCredentials("AccessKeyId","AccessKeySecret",""),S3ForcePathStyle:aws.Bool(true),DisableSSL:aws.Bool(false),}sess:=session.Must(session.NewSession(conf))returns3.New(sess)}
importboto3frombotocore.configimportConfigs3_client=boto3.client('s3',endpoint_url='https://fgws3-gfs.ihep.ac.cn',aws_access_key_id='AccessKeyId',aws_secret_access_key='AccessKeySecret',region_name='us-east-1',use_ssl=True,config=Config(s3={'addressing_style':'path'}),# Path-style access is required)
importsoftware.amazon.awssdk.auth.credentials.AwsBasicCredentials;importsoftware.amazon.awssdk.auth.credentials.StaticCredentialsProvider;importsoftware.amazon.awssdk.regions.Region;importsoftware.amazon.awssdk.services.s3.S3Client;importsoftware.amazon.awssdk.services.s3.S3Configuration;importjava.net.URI;publicclassS3ClientFactory{publicstaticS3Clientcreate(){returnS3Client.builder().region(Region.US_EAST_1).endpointOverride(URI.create("https://fgws3-gfs.ihep.ac.cn")).credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("AccessKeyId","AccessKeySecret"))).serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).build()).build();}}
const{S3Client}=require('@aws-sdk/client-s3');consts3Client=newS3Client({endpoint:'https://fgws3-gfs.ihep.ac.cn',region:'us-east-1',credentials:{accessKeyId:'AccessKeyId',secretAccessKey:'AccessKeySecret',},forcePathStyle:true,});
require'vendor/autoload.php';useAws\S3\S3Client;$s3=newS3Client(['version'=>'latest','region'=>'us-east-1','endpoint'=>'https://fgws3-gfs.ihep.ac.cn','use_path_style_endpoint'=>true,'credentials'=>['key'=>'AccessKeyId','secret'=>'AccessKeySecret',],]);

5.3 Basic Operations

The following examples demonstrate uploading, listing, downloading, and deleting an object. Use an object key in the formdirectory/file, such asdemo/hello.txt. When joining paths, do not let the file name start with/, otherwise//may be generated.

import("bytes""fmt""io""github.com/aws/aws-sdk-go/aws""github.com/aws/aws-sdk-go/service/s3")funcdemo(){svc:=newS3Client()bucket:="your-bucket-name"key:="demo/hello.txt"_,err:=svc.PutObject(&s3.PutObjectInput{Bucket:aws.String(bucket),Key:aws.String(key),Body:bytes.NewReader([]byte("hello gfs")),})iferr!=nil{fmt.Println("upload failed:",err)return}list,_:=svc.ListObjects(&s3.ListObjectsInput{Bucket:aws.String(bucket)})for_,obj:=rangelist.Contents{fmt.Printf("%s %d\n",*obj.Key,*obj.Size)}obj,err:=svc.GetObject(&s3.GetObjectInput{Bucket:aws.String(bucket),Key:aws.String(key),})iferr!=nil{fmt.Println("download failed:",err)return}deferobj.Body.Close()data,_:=io.ReadAll(obj.Body)fmt.Println(string(data))_,_=svc.DeleteObject(&s3.DeleteObjectInput{Bucket:aws.String(bucket),Key:aws.String(key),})}
bucket='your-bucket-name'key='demo/hello.txt's3_client.put_object(Bucket=bucket,Key=key,Body=b'hello gfs')print('upload complete')resp=s3_client.list_objects_v2(Bucket=bucket,Prefix='demo/')forobjinresp.get('Contents',[]):print(obj['Key'],obj['Size'])obj=s3_client.get_object(Bucket=bucket,Key=key)print(obj['Body'].read().decode('utf-8'))s3_client.delete_object(Bucket=bucket,Key=key)print('delete complete')
importsoftware.amazon.awssdk.core.sync.RequestBody;importsoftware.amazon.awssdk.services.s3.S3Client;importsoftware.amazon.awssdk.services.s3.model.DeleteObjectRequest;importsoftware.amazon.awssdk.services.s3.model.GetObjectRequest;importsoftware.amazon.awssdk.services.s3.model.ListObjectsV2Request;importsoftware.amazon.awssdk.services.s3.model.PutObjectRequest;importjava.nio.charset.StandardCharsets;publicclassGfsDemo{publicstaticvoidmain(String[]args){Stringbucket="your-bucket-name";Stringkey="demo/hello.txt";try(S3Clients3=S3ClientFactory.create()){s3.putObject(PutObjectRequest.builder().bucket(bucket).key(key).build(),RequestBody.fromString("hello gfs"));s3.listObjectsV2(ListObjectsV2Request.builder().bucket(bucket).prefix("demo/").build()).contents().forEach(obj->System.out.println(obj.key()+" "+obj.size()));byte[]data=s3.getObjectAsBytes(GetObjectRequest.builder().bucket(bucket).key(key).build()).asByteArray();System.out.println(newString(data,StandardCharsets.UTF_8));s3.deleteObject(DeleteObjectRequest.builder().bucket(bucket).key(key).build());}}}
const{PutObjectCommand,ListObjectsV2Command,GetObjectCommand,DeleteObjectCommand,}=require('@aws-sdk/client-s3');asyncfunctiondemo(){constbucket='your-bucket-name';constkey='demo/hello.txt';awaits3Client.send(newPutObjectCommand({Bucket:bucket,Key:key,Body:'hello gfs',}));constlist=awaits3Client.send(newListObjectsV2Command({Bucket:bucket,Prefix:'demo/',}));for(constobjof(list.Contents||[])){console.log(obj.Key,obj.Size);}constfile=awaits3Client.send(newGetObjectCommand({Bucket:bucket,Key:key,}));console.log(awaitfile.Body.transformToString());awaits3Client.send(newDeleteObjectCommand({Bucket:bucket,Key:key,}));}demo().catch(console.error);
$bucket='your-bucket-name';$key='demo/hello.txt';$s3->putObject(['Bucket'=>$bucket,'Key'=>$key,'Body'=>'hello gfs',]);echo"upload complete\n";$list=$s3->listObjectsV2(['Bucket'=>$bucket,'Prefix'=>'demo/',]);foreach($list['Contents']??[]as$obj){echo$obj['Key'].' '.$obj['Size']."\n";}$file=$s3->getObject(['Bucket'=>$bucket,'Key'=>$key,]);echo$file['Body']."\n";$s3->deleteObject(['Bucket'=>$bucket,'Key'=>$key,]);echo"delete complete\n";

5.4 Usage Tips

  • AccessKeyId and AccessKeySecret are access credentials. Do not write them into public code repositories.
  • A single listing request returns only part of the objects by default. When there are many objects, continue querying with the pagination parameters provided by the SDK.
  • For large files, use the multipart upload capability provided by the SDK.

6 Concepts

gfs is a unified object storage instance built on JWanFS. The system implements strict user-level data isolation to ensure that each user has independent and secure storage space. gfs also removes operating system barriers and supports seamless access from Windows, Linux, and macOS, providing a lightweight and convenient network-disk-like data management experience.

6.1 Bucket

A bucket is functionally similar to the root directory of a familiar file system. It is the top-level container for storing data in JWanFS.

Bucket names are globally unique and cannot be changed after creation.

Naming rules: bucket names can contain only lowercase letters, numbers, and hyphens (-), and must start and end with a lowercase letter or number.

6.2 Endpoint / File Gateway

The endpoint is the entry point for user data access, in the formatIP + port.

gfs currently provides three gateways with load balancing. You can directly usehttps://fgws3-gfs.ihep.ac.cn:

https://fgws3-gfs.ihep.ac.cnhttps://gfs02.ihep.ac.cn:7100https://gfs03.ihep.ac.cn:7100

6.3 AccessKey and SecretKey

Access keys, consisting of AccessKey and SecretKey, are the core authentication credentials for accessing JWanFS storage services. They are used to validate access permissions and protect data access security. Users can obtain key information from theweb management interface.

Appendix

For other usage methods, see the client manual:JWanFS Manual

AI 助手
快速提问 ×

有问题?输入后跳转到对话页继续交流

Baidu
map