Compression Support - Revision history

ShivanSpS: /* LZ41 */

2023-05-12T00:42:51Z

LZ41

@@ Line 11: / Line 11: @@
 The LZ41 implementation is non-LZ4 format standard, just like the official example for random access is, as the LZ4 data format does not seems to consider what is needed for random access.
 FSO checks for the file header on fopen() and if the file is compressed then the logic for compressed files is used.
 == LZ41 Data Format ==

Cyborg17: /* LZ41 Tests */

2023-01-03T15:42:11Z

LZ41 Tests

← Older revision		Revision as of 15:42, 3 January 2023
Line 31:		Line 31:
	<br>		<br>
	'''MVP 4.4.1 - All FS2 Models mission load time'''<br>		'''MVP 4.4.1 - All FS2 Models mission load time'''<br>
			0:50 - NVME Compressed LZ4-HC L6, 7.01GB, BS: 65536<br>
	0:50 - NVME Uncompressed<br>		0:50 - NVME Uncompressed<br>
	~~0:50 - NVME Compressed LZ4-HC L6, 7.01GB, BS: 65536<br>~~
	1:01 - SSD Compressed LZ4-HC L6, 7.01GB, BS: 65536<br>		1:01 - SSD Compressed LZ4-HC L6, 7.01GB, BS: 65536<br>
	1:02 - SSD Uncompressed<br>		1:02 - SSD Uncompressed<br>
Line 38:		Line 38:
	2:30 - HDD Uncompressed<br>		2:30 - HDD Uncompressed<br>
	<br>		<br>

	== LZ41 Compressor Guidelines ==		== LZ41 Compressor Guidelines ==
	'''Block Size'''<br>		'''Block Size'''<br>

Cyborg17: /* Future Improvements */

2023-01-03T15:41:24Z

Future Improvements

← Older revision		Revision as of 15:41, 3 January 2023
Line 73:		Line 73:
	- The extra extension ".lz41" will be added to loose files. Since recent improvements in FSO code allows for loose files to use more than 31 characters. FSO will ignore this extra extension after load and thus "ship.pof.lz41" will be considered as "ship.pof".<br>		- The extra extension ".lz41" will be added to loose files. Since recent improvements in FSO code allows for loose files to use more than 31 characters. FSO will ignore this extra extension after load and thus "ship.pof.lz41" will be considered as "ship.pof".<br>
	- New VP container: .VPC<br>		- New VP container: .VPC<br>
	The VPC container format is identical to a .VP container in structure and limits, with one key difference: It may contain uncompressed files just like any normal VP, but it has to contain at least 1 compressed file.		The .VPC container format is identical to a .VP container in structure and limits, with one key difference: It may contain uncompressed files just like any normal VP, but it has to contain at least 1 compressed file.
	Compressed files stored inside a .vpc container must not have the .lz41 extension or its name changed in any way.<br>		Compressed files stored inside a .vpc container must not have the .lz41 extension or its name changed in any way.<br>
	While .VP can still tecnically be used to store compressed files, it is ~~recomended~~ not to.		While .VP can still tecnically be used to store compressed files, it is recommended not to.
	<br>		<br>
	None of these changes break compatibility with files compressed prior to these changes, but just as with any new feature or major changes, older versions of FSO can not use compressed files in this new way.		None of these changes break compatibility with files compressed prior to these changes, but just as with any new feature or major changes, older versions of FSO can not use compressed files in this new way.

Cyborg17: /* LZ41 Design Choices */

2023-01-03T02:17:28Z

LZ41 Design Choices

← Older revision		Revision as of 02:17, 3 January 2023
Line 66:		Line 66:
	== LZ41 Design Choices ==		== LZ41 Design Choices ==
	A non standart format was used since the official random access example was already breaking the lz4 frame format.<br>		A non standart format was used since the official random access example was already breaking the lz4 frame format.<br>
	Additional data were added trailing to each file, the offset table, the num offsets, the block size, the original filesize, ~~normaly~~ those would be on a container and cached on container load, but it is not possible for loose files.<br>		Additional data were added trailing to each file: the offset table, the num offsets, the block size, and the original filesize. Normally, those would be on a container and cached on container load, but it is not possible for loose files.<br><br>
	Due to the 31 chars filename limit, the idea of adding a extra extension to the files was dropped, as changing the filename ~~lenght~~ was just not an option, and since loose files were getting no extra extension, a uncompressed .pof would be named the same as a compressed .pof~~, as~~ a result ~~of this~~, no need to store compressed files on a ~~diferent~~ container rather than plain .VP was seen as needed as there were already no difference for loose files. This ~~will~~ later ~~been prove~~ to be a mistake because it could cause user ~~confunsion~~.		Due to the 31 chars filename limit, the idea of adding an extra extension to the files was dropped, as changing the filename length was just not an option, and since loose files were not getting an extra extension, an uncompressed .pof would be named the same as a compressed .pof. As a result, no need to store compressed files on a different container rather than plain .VP was seen as needed as there were already no difference for loose files. This was later proven to be a mistake because it could cause user confusion.


	== Future Improvements ==		== Future Improvements ==

Cyborg17: /* LZ41 Compressor Guidelines */ Undo mistake

2023-01-03T02:13:48Z

LZ41 Compressor Guidelines: Undo mistake

Cyborg17: /* LZ41 Compressor Guidelines */

2023-01-03T02:13:26Z

LZ41 Compressor Guidelines

Cyborg17: /* LZ41 Compressor Guidelines */

2023-01-03T02:12:49Z

LZ41 Compressor Guidelines

← Older revision		Revision as of 02:12, 3 January 2023
Line 51:		Line 51:
	.ogg, .wav: .wav may give you some marginal gains, but it is just not worth trying. With ogg its the same problem as with movie files, these files are already compressed.<br>		.ogg, .wav: .wav may give you some marginal gains, but it is just not worth trying. With ogg its the same problem as with movie files, these files are already compressed.<br>
	<br>		<br>
	'''pngs'''		'''pngs'''<br>
	.png and apng: These are already compressed and 99% of the time end up with a negative ratio. pngs that aren't compressed are going to go down in size, considerably, but this is uncommon. It might be possible to check the png header for the compression value they already have before trying to compress them.<br>		.png and apng: These are already compressed and 99% of the time end up with a negative ratio. pngs that aren't compressed are going to go down in size, considerably, but this is uncommon. It might be possible to check the png header for the compression value they already have before trying to compress them.<br>
	<br>		<br>

Cyborg17: /* LZ41 Compressor Guidelines */

2023-01-03T02:12:15Z

LZ41 Compressor Guidelines

← Older revision		Revision as of 02:12, 3 January 2023
Line 40:		Line 40:
	== LZ41 Compressor Guidelines ==		== LZ41 Compressor Guidelines ==
	'''Block Size'''<br>		'''Block Size'''<br>
	Block size is set by the compressor 8192, 16384, 65536 block sizes were all tested, higher values are untested and may work or not~~, a~~ higher block size will make the large compressed files smaller, ~~higher block size~~ will also make FSO to use more RAM ~~but not by much~~.<br>		Block size is set by the compressor 8192, 16384, 65536 block sizes were all tested, higher values are untested and may work or not. A higher block size will make the large compressed files smaller, but will also make FSO use slightly more RAM.<br>
	<br>		<br>
	'''Ignore text based file formats'''<br>		'''Ignore text based file formats'''<br>
	Text files should be ignored~~, they do work~~ fine, but there is ~~no much is~~ to be gained and the only thing it does is to make the file unreadeable by text editors~~: .fc2, .fs2, .tbl, .tbm, .eff~~.<br>		.fc2, .fs2, .tbl, .tbm, .eff: Text files should be ignored. Compressing them works fine, but there is little to be gained and the only thing it does is to make the file unreadeable by text editors. <br>
	<br>		<br>
	'''Movies are already compressed'''<br>		'''Movies are already compressed'''<br>
Line 49:		Line 49:
	<br>		<br>
	'''Audio files'''<br>		'''Audio files'''<br>
	.ogg, .wav: .wav may give you some ~~little~~ gains but it is just not worth ~~to try, with~~ ogg its the same problem as with ~~the movies~~, these files are already compressed.<br>		.ogg, .wav: .wav may give you some marginal gains, but it is just not worth trying. With ogg its the same problem as with movie files, these files are already compressed.<br>
	<br>		<br>
	'''pngs'''		'''pngs'''
	~~pngs~~ are already compressed and 99% of the time end up with a negative ratio~~, and since mods use apngs, you are going to waste time trying to compress them~~. ~~PNG~~ that ~~arent~~ compressed are going to go down in size, considerably, but this is uncommon~~, maybe~~ check the png header for the compression value they already have before trying to compress.<br>		.png and apng: These are already compressed and 99% of the time end up with a negative ratio. pngs that aren't compressed are going to go down in size, considerably, but this is uncommon. It might be possible to check the png header for the compression value they already have before trying to compress them.<br>
	<br>		<br>
	'''Do not keep files that end up with negative ratios'''		'''Do not keep files that end up with negative ratios'''

ShivanSpS: /* LZ41 Data Format */

2023-01-03T01:32:50Z

LZ41 Data Format

← Older revision		Revision as of 01:32, 3 January 2023
Line 13:		Line 13:

	== LZ41 Data Format ==		== LZ41 Data Format ==
	4 byte header (~~0x31345A4C // "14ZL" ->~~ "LZ41"~~, inverted~~) <br>		4 byte header ("LZ41") <br>
	N Blocks<br>		N Blocks<br>
	N offsets<br>		N offsets<br>

Cyborg17: /* LZ41 */ Grammar and spelling

2023-01-03T00:59:36Z

LZ41: Grammar and spelling

← Older revision		Revision as of 00:59, 3 January 2023
Line 7:		Line 7:

	== LZ41 ==		== LZ41 ==
	The LZ41 design is similar to the lz4 random access example~~, it~~ uses the lz4 stream compression, a int table ~~to store~~ the offsets that indicate at what compressed block the original file position is, a int to indicate the number of these offsets, the original filesize and blocksize. The major ~~diference~~ being that instead of using a dictionary, the stream is ~~resetted~~ at each block, making them ~~independient~~ from each other, and thus you can pick any block you want and ~~uncompress~~ that block at will.		The LZ41 design is similar to the lz4 random access example. It uses the lz4 stream compression, with an int table which stores the offsets that indicate at what compressed block the original file position is, an int to indicate the number of these offsets, and the original filesize and blocksize. The major difference being that instead of using a dictionary, the stream is reset at each block, making them independent from each other, and thus you can pick any block you want and decompress that block at will.
	Resetting the stream rather than use a dictionary may be less efficient (this is untested as the time of ~~writting~~ this article), but it was the only way to do it for individual files ~~whiout~~ having to embed the dictionary too.		Resetting the stream rather than use a dictionary may be less efficient (this is untested as the time of writing this article), but it was the only way to do it for individual files without having to embed the dictionary too.
	The LZ41 implementation is non-LZ4 format standard, just like the official example for random access is, as the LZ4 data format does not seems to consider what is needed for random access.		The LZ41 implementation is non-LZ4 format standard, just like the official example for random access is, as the LZ4 data format does not seems to consider what is needed for random access.
	FSO checks for the file header on fopen() and if the file is compressed then the logic for compressed files is used.		FSO checks for the file header on fopen() and if the file is compressed then the logic for compressed files is used.

← Older revision		Revision as of 02:13, 3 January 2023
Line 40:		Line 40:
	== LZ41 Compressor Guidelines ==		== LZ41 Compressor Guidelines ==
	'''Block Size'''<br>		'''Block Size'''<br>
	\|Block size is set by the compressor 8192, 16384, 65536 block sizes were all tested, higher values are untested and may work or not. A higher block size will make the large compressed files smaller, but will also make FSO use slightly more RAM.<br>		Block size is set by the compressor 8192, 16384, 65536 block sizes were all tested, higher values are untested and may work or not. A higher block size will make the large compressed files smaller, but will also make FSO use slightly more RAM.<br>
	<br>		<br>
	'''Ignore text based file formats'''<br>		'''Ignore text based file formats'''<br>
	\|.fc2, .fs2, .tbl, .tbm, .eff: Text files should be ignored. Compressing them works fine, but there is little to be gained and the only thing it does is to make the file unreadeable by text editors. <br>		.fc2, .fs2, .tbl, .tbm, .eff: Text files should be ignored. Compressing them works fine, but there is little to be gained and the only thing it does is to make the file unreadeable by text editors. <br>
	<br>		<br>
	'''Movies are already compressed'''<br>		'''Movies are already compressed'''<br>
	\|.mp4,.ogg,.mve,.webm: the ratio will be negative most of the time, and when is not, you are just saving a few kb, maybe a mb or two.<br>		.mp4,.ogg,.mve,.webm: the ratio will be negative most of the time, and when is not, you are just saving a few kb, maybe a mb or two.<br>
	<br>		<br>
	'''Audio files'''<br>		'''Audio files'''<br>
	\|.ogg, .wav: .wav may give you some marginal gains, but it is just not worth trying. With ogg its the same problem as with movie files, these files are already compressed.<br>		.ogg, .wav: .wav may give you some marginal gains, but it is just not worth trying. With ogg its the same problem as with movie files, these files are already compressed.<br>
	<br>		<br>
	'''pngs'''<br>		'''pngs'''<br>
	\|.png and apng: These are already compressed and 99% of the time end up with a negative ratio. pngs that aren't compressed are going to go down in size, considerably, but this is uncommon. It might be possible to check the png header for the compression value they already have before trying to compress them.<br>		.png and apng: These are already compressed and 99% of the time end up with a negative ratio. pngs that aren't compressed are going to go down in size, considerably, but this is uncommon. It might be possible to check the png header for the compression value they already have before trying to compress them.<br>
	<br>		<br>
	'''Do not keep files that end up with negative ratios'''<br>		'''Do not keep files that end up with negative ratios'''<br>
	\|No matter if those files are inside a vp or they are loose files, if a compressed file end up being bigger than the original one, it must be discarded.<br>		No matter if those files are inside a vp or they are loose files, if a compressed file end up being bigger than the original one, it must be discarded.<br>
	<br>		<br>