Prev B @ Next

Gabe Rocks!

Follow me on the Fediverse at @gabriel@mk.gabe.rocks

Object storage Troubles

I’ve spent some time experimenting with S3 / Object Storage for public content.
I’ve found it useful for things like fediverse attachments & PeerTube videos.

What is object storage?

Object storage is a form of cloud storage that works well to host files cheaply. This is very helpful in circumstances where your own bandwidth is limited and you want to share large files (like video) with the wider internet. You can still back-up your files easily with rclone so you can always change providers if needed.

The snag

When using software that uploads files to object storage. Such as a fediverse instance like PeerTube, it may associate the upload with the storage location in the database. This means that even if you move the files to the new provider, your software will still look for the content in it’s old location. While many software projects support object storage, they may not support migrating storage.

Recommendations

0) Choose a provider carefully

Shop around before choosing a provider. You may be able to find one available at another price. You may not necessarily need object storage depending on what your requirements are, storing files through most applications does work quite well.

1) Don’t change unless you absolutely have to

In my case, switching providers means I have to re-upload my peertube videos if I want the new location to be reflected. This would be a lot of work if I had uploaded a large amount of videos.

2) Where possible, manage files with your own solution

When choosing to support object storage in your project, ensure that the capability to seamlessly migrate providers is a use-case you build for. If you use a static website generator, it can often be quite easy to set your object storage location to a variable and merely update that reference, or simply do a mass “find-and-replace”.

3) Roll your own Object Storage

If I had significant bandwidth, I would setup my own MinIO server that could then be migrated to different hardware or collocation suites if needed.


Reply to this post

Object Storage PeerTube

Gabe Rocks! by Gabriel is licensed under CC BY-SA 4.0