DedupIt 0.2.9 Alpha For test

[Description]
Congratulation Guys!!
Now you can try deduplication for your backups!

DedupIt is a QPKG that allows you to deduplicate shared folders.
Deduplication reduce the space usage when storing similar files, data, such as ghosts, backups, VHDx dump, SQL dump,...
It is designed and optimized for Backup and Archiving.
The deduplication process is offline, and in the background, following your schedule.
You can enable deduplication on shares, and disable it later. When you disable it on a share, the content of the share will be expanded to its original size (this takes time).

This QPKG is ALPHA version for test and not ready for production environments.
My friend and I have tested it and used it for several months without big issues so far.

The UI should be intuitive enough.
Please feel free to give any comments.
Also, if you have some test results, feel free to post them in the thread, or send me the results by PM.




[Disclaimer]
DedupIt, the software, is provided for testing purposes only. This software is supplied "AS IS" without any warranties and support. We, the developers, assume no responsibility or liability for the use of the software. We reserve the right to make changes in the software without notification. We also make no representation or warranty that such application will be suitable for all environments without further testing or modification.


[Recommendations]
- DO NOT USE ON PRODUCTION ENVIRONMENTS
- Use for testing purpose only
- Use for backup files and archives only
- Create a separate shared folder to test this QPKG.
- Test on powerful NAS device. x79 and above is recommended. However, small NAS should work, but the deduplication process may need much more time for lower results.
- Do not use compression when backing up to the deduplicated folder. Compression will reduce the deduplication efficiency… A LOT.
- Media files, such as pictures, videos… because of the file content type, it is known that the result will not be good. Even if 2 pictures looks the same, the content of the file will be totally different, which reduce the deduplication ratio.
- Do not use it on normal shared folders (such as where you have your Word documents). The read performance decreases too much for a comfortable use. It is designed for backup and archive.
- Keep the default file modified time to 1 day to avoid trying to deduplicate a file being modified. Opened files and modified files during the deduplication process are skipped. However, it may take additional time if the file is modified while being deduplicated.


[Limitations]
- Supports x86 NAS Only, with 4.2.0 Beta Firmware
- Hardlinks are not currently supported so Backup Versioning QPKG cannot be used because I believe it uses harlinks. (This limitation should be resolved in a near future).
- The first time deduplication still need to be optimized. If possible, use only “unique data” for the first time deduplication. It will improve the further results when adding files to the deduplicated folder.
- Encrypted volumes and encrypted shares are not supported.
- Default Shares are not supported to avoid side effect we may not know.
- Read performance of deduplicated files are much slower because of the fragmentation (normal and not easy to improve).
- Deduplicated files can be modified. However, they will not be deduplicated again. (to do list)


[Known issues]
- Hardlinks are not currently supported so Backup Versioning QPKG cannot be used because I believe it uses harlinks. (This limitation should be resolved in a near future).
- Encrypted shares, shares on encrypted volumes and iso shares are not filtered. Do not add those shares to DedupIt.
- Cannot create a Vault extension if there is no deduped folder. This cause an unexpected behavior, do not do it.
- Cannot increase a Thick Provisioning Vault in some condition
- Move a vault extension to the same folder as the original location will corrupt the configuration.


[FAQ]
Q: Can I remove the QPKG and keep my data?
- If you remove the QPKG, the data in the deduplicated folder will become inaccessible. If you reinstall the QPKG immediately, your previous configuration will be restored if all files and shares are still available.
- If you remove the QPKG and remove the vault shared folder, the setup wizard will prompt you to setup the QPKG from the beginning.
- If you stop the QPKG, the deduped shared folders will become inaccessible
- If you delete the vault, or part of the vault, all the deduplicated data will be lost

Q: How to remove the QPKG correctly?
- To remove the QPKG without losing deduplicated data, disable deduplication on each shared folder first, until no more shares are in the list, then the QPKG can be removed.

Q: I’m trying to test with some files but it always says there is no new files matching the deduplication policy.
- Even if “modified time” is set to 0, there is a 5 minutes delay before a file can be deduplicated. It is hardcoded and cannot be changed.


[Release Notes]
v0.2.9 - Initial Alpha Test Release