Slow PNG encode? Memory stick issue?

[bulb]

Hi,

I am new to homebrewing on PSP. I have written the code that takes a snapshot of current display and saves it as PNG.

I have compiled libpng 1.2.8 and zlib 1.2.3 without any modification to the source code.

When I take a snapshot it takes approx 10 seconds to finish the job. I thought PSP is faster than this.

I believe access to memory stick is slowing things down. I did some debugging - tracing to file when each scan line gets processed by libpng. The log function is simple:

void Log(const char *s)
{
FILE *fp = fopen("trace.txt", "a");
fputs(s, fp);
fclose(fp);
}

The encoding process takes 2 and a half minutes when the logging is turned on!!

Is memory stick really that slow? Any suggestions how to speed up the encoding process?

Thanx

[Psilocybeing]

I noticed the exact same thing when taking screenshots directly from VRAM. Just a thought, but could copying the contents of VRAM to memory first, and THEN writing to a PNG speed things up?

[bulb]

Psilocybeing,

I doubt this is an issue, because I convert each scan line before calling libpng. The converting is almost identical to copy operation.

Is reading VRAM slower than writting to it?

Nevertheless, this doesn't address outrageous performance hit with logging.

Thanx

[miemt11]

Originally Posted by bulb

Hi,

I am new to homebrewing on PSP. I have written the code that takes a snapshot of current display and saves it as PNG.

I have compiled libpng 1.2.8 and zlib 1.2.3 without any modification to the source code.

When I take a snapshot it takes approx 10 seconds to finish the job. I thought PSP is faster than this.

I believe access to memory stick is slowing things down. I did some debugging - tracing to file when each scan line gets processed by libpng. The log function is simple:

void Log(const char *s)
{
FILE *fp = fopen("trace.txt", "a");
fputs(s, fp);
fclose(fp);
}

The encoding process takes 2 and a half minutes when the logging is turned on!!

Is memory stick really that slow? Any suggestions how to speed up the encoding process?

Thanx

It might be better, if you post your question here

PSPDev
http://forums.ps2dev.org/viewforum.php?f=14

It is a good site to start understand PSP deveop

[Garak]

Hi,

I'm not sure, but from the looks of your code you are passing in a single charcter, opening a file, writing the charcter, then closing the file. IFF this is the case, writing only a single charcter of output at a time is bad enough, but opening and closing the file each time takes the taco. That process would be noticibly slow on even a Cray Super Computer or something.

The best way to speed things up, if of course to open the file 1 time, and write as much data as you can to the file in 1 sitting. That should greatly increase your performance.

Garak

[bulb]

Garak,

You should take a better look at the code snippet next time. I am actually logging one string at a time. I rather append to the file, because this is how real logging should be done. Writting to the file is buffered and crashing the application will not display cached logs. I could use fflush() to flush the buffers, but it didn't do the trick (i have tried).

Almost all tracing in production code (being games or top notch applications) use logging like this. I have actually seen such code in numerous applications and you don't need a Cray to run it.

Anyway, I am more interested why the actual code is so slow (w/o logging).

Bulb

[lord pip]

can i try your homebrew?
sneaky way to become a beta tester, but if might be just your psp.
also have you tried outputting a log file that tells you "how long" "for "each step"?
It could be a memory stick issue.

[bulb]

lord pip,

That would be great! Tell me how to send you the file. I am working on FW 1.5 btw.

I haven't logged any time information, because I don't know how to aquire such information yet. Though quitting the "game" at different time gave me enough information.

Thanx

[bulb]

I've done simple test now that just writes 100 Kbytes to memory stick. I have written it:

a. in one single block and
b. one byte at a time (using fwrite() 100000 times)

Both methods took neary the same time. The problem is that they take about 40 seconds to complete! This is 2.5 Kb/sec.

Is this rate OK or is there something wrong with my memory stick or (hopefully not) with my PSP?

If this is the normal rate, then I guess I should better move my file I/O to different thread (assuming that main thread can run, when worker thread is blocked with I/O operation). Currently I only use one thread. Could this be my sole problem?

Thanx

[Garak]

When I looked up the def of fputs, my eyes strayed to fputc, which only writes 1 charcter. I see that fputs does in fact write a string. But wait... that should be even worse.

fputc will write everything in your string up to the \0 (NULL terminator) to the end of the file. If this is PNG data, then it is binary data, and it is perfectly acceptable for the file to contain null byte charcters, which means you would want them coppied to the file. I don't think this code would even write the PNG to the file correctly. Have you tried opening one of these saved files? Or is there more going on behind the scenes in other functions that I am not aware of? All I have to go on is your 1 sample.

As for opening/closing the file each time, do what you want. But obviously it wil be faster if you open/close the file only 1X. Sure, continually opening/closing is fine when you are "Debugging" and writing error messages to file. But if you do something simple and controlled like a copy from memory to a file, there is no need to continaully open/close. The odds of crashing during this time are slim, and any output given to the file in the event it does crash will be pretty useless to you anyway.

There are allready examples of how to load/save screenshots using the PNG library also. That might be a good place to start looking as well. Have you checked any of this stuff yet on your Cray to see if it t least runs there ?


Garak

[bulb]

Garak,

I don't understand what you are trying to tell about fputc(), since I am not using fputc(), fputs() accidently use fwrite().

Obviously I didn't use Log() function to write PNG files, it is used solely for printing lines to trace file. Who ever gave you other idea? I said i didn't touch libpng sources, which use fwrite() to write buffer to stream.

You should better have more experience with C/C++ before writting witty comments.

Better yet, look at my last post, which describes what the real problem is.

Bulb

[Garak]

Oh,

Oh, I'm sorry. I see what the problem is now. You're a moron.

Garak

[bulb]

Everyone,

I was looking through some example code and noone uses another thread for file IO, so this must not be an issue.

I experimented with another memory stick, which proved faster: doing 100 Kb writes in 7 seconds, instead of 40 seconds. Still way too slow. Interesting note: both memory sticks are from Sony and have 1 Gb capacity.

I was using slower memory stick in further examination.

Original code giving 2.5 Kb/s rate:

u8 buffer[100000];
FILE *f = fopen("test.bin", "wb");
fwrite(buffer, 100000, 1, f);
fclose(f);

Modified code giving 150 Kb/s rate:

u8 buffer[100000];
int f = sceIoOpen("test.bin", PSP_O_WRONLY | PSP_O_CREAT, 0777);
sceIoWrite(f, buffer, 100000);
sceIoClose(f);

Seems using Sony's file IO functions directly gives much better performance than C runtime library. This seems strange, because CRT is just a wrapper around Sony's file IO. Seems like a bug of CRT. Sadly I can't check the sources, because I am using devkitpro R6, which comes precompiled.

Newly released MSTest v1.0 also gives such transfer rate, with various buffer sizes it was possible to squeeze it to max 170 Kb/s.

I am still not completly satisfied though, because USB copy to memory stick gives me around 2500 Kb/s. Though, I have run out of ideas where to get better transfer rate.

I would really like to hear your experience with write transfer rate.

[Shatterdome]

It's very possible that that is the internal max that it can write to (from memory, or otherwise)....did you even attempt to try writing to it by keeping the file open and just closing it when you are done ? instead of opening and closing everytime ? Even if opening and closing takes a 10th of a second, that's still alot of time when you do it 100,000 times :P Because your second example, using the sceIO functions are about as good as it gets, unless you do happen to have a defective PSP

In my experience any program that takes a screenshot freezes my psp for 2-3 seconds, saving and loading games can take awhile too, but with a USB2, like you said you get really good transfer rates....so maybe it has something to do with writing from the usb or from the system memory...

It sounded like you tried that Memory stick tool that was just released ? If not you should give it a shot and see what your rates are....maybe ill try it a little later on as well (I also have a sony MS), but if that guys program was able to induce 4-5 mbs transfer rates, there's no reason your's shouldn't either....if you use his program and still get 150kbs transfer rates, then you know it's your hardware (Sony's cards are slower then sandisk as well)

Anyways, good luck in finding out whats the problem...

[bulb]

Shatterdome,

I wasn't opening and closing the file when doing real IO, just when I was doing debug tracing. This is not an issue in later posts. As you can see, the 1st example has the same functionality as the 2nd example (CRT vs Sony IO).

Anyway, I don't have an issue with having Sony's slower memory stick, the issues here are:

1. Why is CRT IO up to 70 times slower than Sony's IO? Seems like a bug in CRT code.

2. Why are Sony's IO routines 14 times slower than using USB transfer? Is this deliberate action of Sony? Developers would not need fast transfer rates for writing and reading saves, so did Sony choose to put some waits in these routines? If so, is it possible to get around these waits?

What I would really like to see if someone tries to compare USB transfer rate versus transfer rate of his own code.