Storage

Introduction

Cosmos proposes multiple ways to store and manage your data. Before diving into the different storage options, it is important to understand the different types of storage that Cosmos uses, and when to use which. It is important to note that aside from RAID, all those options can be used without even formatting your disks, it is extremely flexible and always allow you to see all your indidual files in each disk. This setup is described here

• Direct Storage: If you just plug-in a disk and start saving data into it. It is the simplest, and works great if you have < 2 disks, and < 1TB of data. In that scenario external backups will allow you to recover your data.
• Parity Disks: If you have more disks/data, you might want to use parity disks. This will allow you to recover your data if a disk fails, without having to re-download a heavy backup from somewhere. It is a good option if you have < 5 disks, and < 10TB of data. The way it works is that you have a parity disk, which is a disk that contains a "parity" (some maths stuff) of all the other disks. If one disk fails, you can recover the data from the parity disk. This is not a backup, but it is a good way to protect your data. If you have 1 parity disk, you can recover from 1 disk failure, if you have 2 parity disks, you can recover from 2 disk failures, etc. More details here
• MergerFS: This is best used in situation where you have many disks, it allows you to access them under one folder. This can be used in conjunction with Parity disks. It is a good option if you have < 10 disks, and < 20TB of data. It is a good way to manage your data, and to have a good overview of your storage. more details here
• RAID: If you have a lot of disks, and a lot of data, you might want to use RAID. RAID is a way to store data on multiple disks, and to recover from disk failures. It is a good option if you have > 6 disks, and > 10TB of data. It is the best way to protect large amount of data, with the best performance. More details here. We recommend using RAID 6, as it allows you to recover from 2 disk failures.

Direct Storage Management

Format

Cosmos allows you to format your disks in different formats. The most common format is ext4, but you can also use NTFS, FAT32, exFAT, and XFS. It is recommended to use ext4, as it is the most stable and reliable format. It is also the most supported format in Linux. You can also use NTFS, FAT32, exFAT, and XFS, but they are not as stable and reliable as ext4. You can also use multiple formats on the same disk, but it is not recommended, as it can cause issues.

You will notice the format dialog does not let you customize the partition. This is because it is highly recommended you do not partition your disks. It is best to use the whole disk, as it will make it easier to manage your data. If you need to partition your disk, you can do so using the command line. Cosmos will create a single partition on the disk, and format it with the selected format.

S.M.A.R.T.

Cosmos allows you to check the S.M.A.R.T. status of your disks. S.M.A.R.T. is a way to check the health of your disks. It is recommended to check the S.M.A.R.T. status of your disks regularly, as it will help you detect disk failures before they happen. You can check the S.M.A.R.T. status of your disks by going to Storage > Disks > Select your disk > S.M.A.R.T.

More info on S.M.A.R.T.

Parity Disks / SnapRAID

Parity disks are a way to protect your data from disk failures. It is not a backup, but it is a good way to protect your data. The way it works is that you have a parity disk, which is a disk that contains a "parity" (some maths stuff) of all the other disks. If one disk fails, you can recover the data from the parity disk. This is not a backup, but it is a good way to protect your data. If you have 1 parity disk, you can recover from 1 disk failure, if you have 2 parity disks, you can recover from 2 disk failures, etc.

The parity disks in Cosmos are snapshot based: They do not get calculated in real time, instead, they get "sync'd" on a regular basis. When dealing with a few TB of data, this is the best options. Running real time parity is very costly and will slow down your system. The sync process is very fast, and can be scheduled at a time where you do not use your system. If you have more data, you can use RAID which is a much better option than parity disks. being snapshot based, you can recover from any point in time, not just the last sync.

Setup

To setup Parity Disks, you need to have a few things:

• A few disks: You need at least 3 disks, two for the data, and one for the parity. You can have more disks, but you need at least 3.
• A parity disk: This is a disk that will contain the parity of all the other disks. It is recommended to have 1 parity disk, but you can have more if you want to protect against more disk failures. We recommend using a single Parity disk.

To setup Parity Disks, you need to follow these steps:

• Go to Storage > Parity > New

• Follow the steps in the modal, first select a Parity disks. This has to be a disk that is not used for data, and it has to big the same size or bigger than the biggest data disk.
• Then select the data disks, you can select multiple disks. You will notice that disks have a name: disk 1, disk 2. This is important when recovering data.
• You can customize the Sync / Scrub schedule. By default it is set to sync every day at 2am, and scrub every 2 days at 4am. You can change this to your liking.
• click on "Create".

Sync

The sync process is the process of calculating the parity of all the data disks. This is a very fast process, and can be scheduled at a time where you do not use your system. It is recommended to sync at least once a day, but you can do it more often if you want. The sync process is very fast, and can be scheduled at a time where you do not use your system.

You can manually run a sync or check the logs of each sync in the Scheduler tab:

Scrub

The scrub process is the process of checking the integrity of the data. It is a very important process, as it will detect any corruption in your data. It is recommended to scrub at least once a week, but you can do it more often if you want. The scrub process is very fast, and can be scheduled at a time where you do not use your system.

You can also manually run a scrub or check the logs of each scrub in the Scheduler tab.

Recover Data (Fix)

If a disk fails, you can recover the data from the parity disk. To do so, you need to follow these steps:

• First, we need to replace the broken disk with a new one: Go to Storage > Parity > Select your storage > Edit.
• In this screen, you will see one of the disk is missing. Select a new disks in the dropdown to replace it:

• Click on "Save"
• Then, go to Storage > Parity > Select your storage > Fix. Wait for the process to finish, and your data will be recovered.

MergerFS

MergerFS is a way to access multiple disks under one folder. It is a good way to manage your data, and to have a good overview of your storage. It is best used in situation where you have many disks, and you want to access them under one folder. This can be used in conjunction with Parity disks. It is safe to remove a disk from MergerFS, as it will not delete the data on the disk. It is also safe to remove MergerFS, as it will not delete the data on the disks.

Setup

To setup MergerFS, you need to have a few things:

• A few disks: You need at least 2 disks, but you can have more if you want. You can have different size disks, but it is recommended to have the same size disks.
• A mount point: This is the folder where you will access all your disks. It is recommended to have a dedicated folder for this, and not to mix it with other data.

To setup MergerFS, you need to follow these steps:

• Go to Storage > MergerFS > New

• Follow the steps in the modal, first select a mount point. This is the folder where you will access all your disks. It is recommended to have a dedicated folder for this, and not to mix it with other data.
• Then select the disks, you can select multiple disks. You can customize the options, but the default options are good for most users.
• click on "Create".

Other options

Permanent Mounts

You will notice that those settings allow you to create permanent mounts. This is a good way to mount your disks, as it will survive a reboot. It is recommended to use this option, as it will make sure your disks are always mounted.

Chown

By default, the disks are mounted as root. This is not a good idea, as it will make it hard to access your data. You can change the owner of the disks by using the chown option. This is a good way to make sure you can access your data.