Improve compression

stephenseo commented

2024-09-24 08:02:08 +00:00

Owner

SimpleArchiver incurs a large amount of overhead-data when each file is compressed individually. One idea is to concatenate files into chunks and compress the chunks for less overhead. Perhaps a default minimum size per chunk could be 1 MiB, but some tuning/tweaking may be required to optimize it.

stephenseo added the

Kind/Enhancement

label 2024-09-24 08:02:08 +00:00

stephenseo self-assigned this 2024-09-24 08:02:08 +00:00

stephenseo referenced this issue from a commit

2024-09-24 09:47:08 +00:00

Create file format for format version 1

stephenseo commented

2024-09-24 09:50:54 +00:00

Author

Owner

https://git.seodisparate.com/stephenseo/SimpleArchiver/src/branch/issue_18_improve_compression will track the progress of this feature until it is merged.

stephenseo added

Kind/Feature

and removed

Kind/Enhancement

labels 2024-09-24 09:51:49 +00:00

stephenseo added this to the Development Planning project 2024-09-24 09:59:30 +00:00

stephenseo referenced this issue from a commit

2024-09-26 08:34:54 +00:00

Create file format for format version 1

stephenseo commented

2024-09-27 06:56:50 +00:00

Author

Owner

As of c71f4f45c7 basic extraction is implemented. Uncompressed chunked files can be extracted. Extracting symlinks is implemented but untested. What's left is to test extracting symlinks, extract compressed chunked files, and to create archive files in the new format.

As of https://git.seodisparate.com/stephenseo/SimpleArchiver/commit/c71f4f45c7b49381d9641033a30a320e7be31c0c basic extraction is implemented. Uncompressed chunked files can be extracted. Extracting symlinks is implemented but untested. What's left is to test extracting symlinks, extract compressed chunked files, and to create archive files in the new format.

stephenseo referenced this issue from a commit

2024-09-28 10:09:35 +00:00

Create file format for format version 1

stephenseo commented

2024-10-04 08:23:23 +00:00

Author

Owner

This feature is about done, just needs some testing to verify it works correctly.

stephenseo commented

2024-10-07 04:29:05 +00:00

Author

Owner

There were some errors de/compressing archives when using large files. The code has been updated to be more robust and some testing indicates that it works now. Still will test more.

stephenseo referenced this issue from a commit

2024-10-08 03:43:58 +00:00

Merge branch 'issue_18_improve_compression'

stephenseo closed this issue

2024-10-08 03:43:59 +00:00

Improve compression #18