Author |
Message |
k0zy Server Help Squatter
Gender: Joined: Jan 11 2003 Posts: 571 Location: Germany Offline
|
Posted: Fri Sep 01, 2006 3:44 am Post subject: JPG/PNG Compression |
|
|
|
|
As you might know I'm coding a LvzViewer.
So today I started to decompress files.
As the lvzFormat.txt states how to handle the *data* part: Quote: | Read this all in, and then run it through zlib's decompress function. |
That's what I did, when ZipLibSharp gave me an error message about missing zlib header.
JPGs and PNGs are not compressed at all in the lvz.
Why?
Even if it's just a few byte, it would result in smaller lvz files.
Did you forget to put that information in lvzFormat.txt, MGB?
Did PriitK forget to compress them?
Maybe it's forbidden to compresses them?
Or did just nobody notice?
DeBuildLevel handles the files just fine... Did zlib not complain about the header? _________________ It's a shark! Oh my god! Unbelievable! |
|
Back to top |
|
|
Mine GO BOOM Hunch Hunch What What
Age:40 Gender: Joined: Aug 01 2002 Posts: 3614 Location: Las Vegas Offline
|
Posted: Fri Sep 01, 2006 3:57 am Post subject: Re: JPG/PNG Compression |
|
|
|
|
Bob Dole.. Bob Dole... Bob Dole...... bob dole.... bob... dole.... wrote: | That's what I did, when ZipLibSharp gave me an error message about missing zlib header.
JPGs and PNGs are not compressed at all in the lvz. |
You need to use whatever is direct access to zlib's compress/decompress functions, not zip wrappers. There is no header info for it. It is a raw compressed stream, the same as .gz files. I've never used C# before, so I can't say. Google says that there are tons of wrappers out there, but for SharpZipLib, it looks like you should be using GZipInputStream for decompression. There is sample code they supply that looks like what you need (gzip, not zip).
As for the compression not being there, if you look at the FileTime field, if it is zero, there is no compression. Its weird, but thats how Continuum handles it, so thats how I made DeBuildLevel. You can have anything you want compressed, but the reason it isn't compress is that either the output stream is greater than the input (ie: you can't compress compressed data forever) or the time it would take to decompress the file isn't worth the time so it isn't compressed (ie: jpg/png usually). This is all upto the program that creates the LVZ file, because if you want to put a compressed file that is 1000 times larger the the original file, thats supported. |
|
Back to top |
|
|
k0zy Server Help Squatter
Gender: Joined: Jan 11 2003 Posts: 571 Location: Germany Offline
|
Posted: Fri Sep 01, 2006 4:32 am Post subject: Re: JPG/PNG Compression |
|
|
|
|
Mine GO BOOM wrote: | [..]
You need to use whatever is direct access to zlib's compress/decompress functions, not zip wrappers. There is no header info for it. It is a raw compressed stream, the same as .gz files. I've never used C# before, so I can't say. Google says that there are tons of wrappers out there, but for SharpZipLib, it looks like you should be using GZipInputStream for decompression. There is sample code they supply that looks like what you need (gzip, not zip).
As for the compression not being there, if you look at the FileTime field, if it is zero, there is no compression. Its weird, but thats how Continuum handles it, so thats how I made DeBuildLevel. You can have anything you want compressed, but the reason it isn't compress is that either the output stream is greater than the input (ie: you can't compress compressed data forever) or the time it would take to decompress the file isn't worth the time so it isn't compressed (ie: jpg/png usually). This is all upto the program that creates the LVZ file, because if you want to put a compressed file that is 1000 times larger the the original file, thats supported. |
I don't understand.
I just looked at the data I read from the lvz again.
The only time the filetime is zero is when I read the objects.
I'm only able to tell, that the file has not been compressed if both sizes, before and after compression, are the same.
I know you can't compress compressed data forever.
However, the most PNGs and JPGs I found within lvz files compress just fine (I tried, they did get smaller).
And even if they get larger, it's not like 1000 times
I understand that in this case you rather store the original file.
I don't think gzip is able to decompress zip, because it's a whole diffrent algorithm. (Read gzip homepage)
And ZiplibSharp decompresses the files just fine with it's zlib port (not gzip)!
It doesn't give the "missing header" error on compressed files (all but jpg/png)! So this header is maybe where the dictionary is stored, that'd mean that even a raw stream would have that header.
I can decompress bmps, hlps, bm2s, but not png/jpg, because they simply aren't compressed, this has nothing to do with ZiplibSharp.
If you search google for C# and zlib the only things you find are:
zlib wrappers, and forum posts where users try to get zlib1.dll working but then just give up and rather use a wrapper.
EDIT: Reading your post again I think you misunderstood something.
ZipLibSharp is able to open zip files, but that's not what I used it for and I didn't use it's zip file methods. |
|
Back to top |
|
|
Bak ?ls -s 0 in
Age:24 Gender: Joined: Jun 11 2004 Posts: 1826 Location: USA Offline
|
Posted: Fri Sep 01, 2006 10:44 am Post subject: |
|
|
|
|
yeah, when I was making X-LvzToolkit I relized that lots of times the compressed size > uncompressed size, so in that case I leave it uncompressed. Another giveaway that it's uncompressed is if compressed size == uncompressed size (i think continuum looks for this, too).
In either case, if you think it's uncompressed, just output it to a file and see if you can open it in a picture editor. _________________ SubSpace Discretion: A Third Generation SubSpace Client |
|
Back to top |
|
|
k0zy Server Help Squatter
Gender: Joined: Jan 11 2003 Posts: 571 Location: Germany Offline
|
Posted: Fri Sep 01, 2006 1:03 pm Post subject: |
|
|
|
|
Bak wrote: | yeah, when I was making X-LvzToolkit I relized that lots of times the compressed size > uncompressed size, so in that case I leave it uncompressed. Another giveaway that it's uncompressed is if compressed size == uncompressed size (i think continuum looks for this, too).
In either case, if you think it's uncompressed, just output it to a file and see if you can open it in a picture editor. |
Which makes perfect sense.
It just seems to me that in some cases the png and jpg could be even smaller...
And this should be in lvzFormat.txt
Bob Dole.. Bob Dole... Bob Dole...... bob dole.... bob... dole.... |
|
Back to top |
|
|
Smong Server Help Squatter
Joined: 1043048991 Posts: 0x91E Offline
|
|
Back to top |
|
|
grazzhoppa Novice
Joined: Jan 03 2007 Posts: 29 Offline
|
Posted: Wed Jan 03, 2007 8:42 pm Post subject: |
|
|
|
|
How should these be handled:
File time is zero, name length is more than 0
---- bad section header? Should file times be guarenteed to be not zero when there is a file name?
File time is not zero, name length is 0
---- bad section header? The file does not have any name.
CLT and smong's unfinished lvz reader only check for, as I could see:
if(File Time being 0 and File Name length being 0)
to distinguish between the section being a file or map/screen object info. |
|
Back to top |
|
|
Bak ?ls -s 0 in
Age:24 Gender: Joined: Jun 11 2004 Posts: 1826 Location: USA Offline
|
Posted: Wed Jan 03, 2007 9:30 pm Post subject: |
|
|
|
|
Have you seen lvzFormat.txt?
After this, if the count is > 0, there should be a compressed header:
4-len str Type - Should be 'CONT' also. If not, the compression header is bad, and do not use.
i32 Decompress Size - The size of the data after it would be decompressed. This is the value you should set your out-buffer for the zlib's compression call.
i32 File Time - This can either be a file time, if this compressed section is a file, or it could be 0. If 0, then is a special compressed section. The file time is the standard time_t format used by windows/unix.
i32 Compressed Size - The size of the compressed data in the .lvz. This is how many bytes you'll need to read to get your required data.
Null-end str File Name - This string is always null ended. If no file time, the File Name should be blank (just the 0x00), but do not always count on it.
* data * Compressed data - This should be the length of the Compressed size. Read this all in, and then run it through zlib's decompress function.
...
The decompressed data can be 1 of two types for the current .lvz format. The first type is more often found in .lvzs, and this is the file type. Files are determined by having a File Time not equal to 0, and having a File Name length not being 0. Please be warned, future formats of .lvz's may change one or the other of these checks, so a good safety feature would be that files require both of these to be true. Files are normally just saved to the proper folder and the correct file time set to it. If you do not know how to set file times, its is not 100% important for most DeBuildLevel like programs.
The only other current decompressed data type is the Object section. The object section is determined by having a File Time equal to 0 and the File Name length being 0.
|
I think the spec is silent about those two cases you mentioned. |
|
Back to top |
|
|
grazzhoppa Novice
Joined: Jan 03 2007 Posts: 29 Offline
|
Posted: Wed Jan 03, 2007 9:52 pm Post subject: |
|
|
|
|
That's why I'm asking for advice on it - its not defined
Should I try to extract the other sections in the .lvz when I encounter one of those situations or assume the whole file is bad?
Would a programmer who has made an .lvz creator ever allow those conditions to be written into an .lvz?
There might be a bug in some .lvz creation program that puts empty file names into the lvz. If this is true, then every section in the lvz except the one with an empty file name could still be good.
I've never written things that decode someone else's specification, and I'm asking for advice about handling those situations. |
|
Back to top |
|
|
Bak ?ls -s 0 in
Age:24 Gender: Joined: Jun 11 2004 Posts: 1826 Location: USA Offline
|
Posted: Thu Jan 04, 2007 12:43 am Post subject: |
|
|
|
|
print an error and quit or keep parsing... it's your choice, chances are you'll never come across an LVZ file like that anyway. |
|
Back to top |
|
|
Animate Dreams Gotta buy them all! (Consumer whore)
Age:36 Gender: Joined: May 01 2004 Posts: 821 Location: Middle Tennessee Offline
|
Posted: Thu Jan 04, 2007 2:29 am Post subject: |
|
|
|
|
I was wondering, what language would this be coded in? And if it's not a cross-platform langugae, are you going to release a linux version? I was just thinking yesterday how much I would love one of these on Linux. >_> |
|
Back to top |
|
|
Cyan~Fire I'll count you!
Age:36 Gender: Joined: Jul 14 2003 Posts: 4608 Location: A Dream Offline
|
Posted: Thu Jan 04, 2007 11:54 am Post subject: |
|
|
|
|
As it says in the linked post, C#. Which means a no-go for a Linux version.
(Well, I guess you could try Mono?) _________________ This help is informational only. No representation is made or warranty given as to its content. User assumes all risk of use. Cyan~Fire assumes no responsibility for any loss or delay resulting from such use.
Wise men STILL seek Him. |
|
Back to top |
|
|
Bak ?ls -s 0 in
Age:24 Gender: Joined: Jun 11 2004 Posts: 1826 Location: USA Offline
|
Posted: Thu Jan 04, 2007 4:44 pm Post subject: |
|
|
|
|
what's wrong with X-LvzToolkit Animate? |
|
Back to top |
|
|
D1st0rt Miss Directed Wannabe
Age:36 Gender: Joined: Aug 31 2003 Posts: 2247 Location: Blacksburg, VA Offline
|
Posted: Thu Jan 04, 2007 8:15 pm Post subject: |
|
|
|
|
A lot of stuff will run under mono just with a simple recompile (provided you aren't referencing any binaries that you don't have the source to), though some minor modifications might need to be made _________________
|
|
Back to top |
|
|
Animate Dreams Gotta buy them all! (Consumer whore)
Age:36 Gender: Joined: May 01 2004 Posts: 821 Location: Middle Tennessee Offline
|
Posted: Fri Jan 05, 2007 4:59 am Post subject: |
|
|
|
|
Hmm, Bak, the biggest problem is having not heard about it before. Granted, I didn't do any searching - I just assumed that since he was making one, that one didn't already exist. I'll try this out in a bit. |
|
Back to top |
|
|
|