background
A meteorological data display project done by the company involves a lot of data analysis in the original format, such as grid meteorological data. The internal data is generally a two-dimensional array, which is generally stored in binary mode.
The essence of grid data can be understood as a picture, and each pixel has a data value.
task
I took over the parsing work to convert the data in binary format into a more general text format for easy viewing and display.
Another additional task is that the density of the original data is high, i.e. 1km × 1km, 435km in the vertical direction and 355km in the horizontal direction, involving a total of 435x355=154425 points. Due to the lack of awesome in front-end optimization, the density can only be degraded to 5km × 5km.
Parsing code
The so-called binary data is to code the data one by one, so the code is not difficult. It can be completed by knocking for a while:
// InputStream in; Stream object float[][] data = new float[HEIGHT][WIDTH];//Receive data using float array byte[] buf = new byte[2];//buf BufferedInputStream bin = new BufferedInputStream(in);//Using cache stream objects for(int y=0; y<HEIGHT; y++) {//Read line by line from the top left for(int x=0; x<WIDTH; x++) { data[y][x] = read(bin, buf);//Read a data } skip(bin,WIDTH*5*2*4);//Skip 4 lines down }
problem
The data looks like this:
After I parsed it, I put the results into the display interface to view. The parsed results are striped data.
And the crunching thing is:
But if I don't downgrade the density, it's normal again (analyze the original width and height one by one without skipping).
process
In the middle is a painful trial and error process. I try to print the position of the current stream, because this shape looks like a dislocation. However, I record it through a position and find it is normal.
Turnaround
In the process of passing, I told myself from time to time that there must be something wrong with what was written.
By chance, when I created the cache stream, I added a parameter size=3550 (just the size of 5 rows)
BufferedInputStream bin = new BufferedInputStream(in, 3550);
Ah!
That's what I want.
It reminds me of that song - "my skateboarding shoes".
reason
In fact, it's very simple. The problem lies in
skip(bin,WIDTH*5*2*4);//Skip 4 lines down
And I wrote it like this (to avoid throwing checkedException)
try { inputStream.skip(n); }catch (Exception ex){ throw new RuntimeException("reading data error:", ex); }
In other words, I think this skip will ensure that the number of bytes needed to skip is actually skipped, and then find the skip method of BufferedInputStream
public synchronized long skip(long n) throws IOException { this.getBufIfOpen(); if (n <= 0L) { return 0L; } else { long avail = (long)(this.count - this.pos); if (avail <= 0L) { if (this.markpos < 0) { return this.getInIfOpen().skip(n); } this.fill(); avail = (long)(this.count - this.pos); if (avail <= 0L) { return 0L; } } long skipped = avail < n ? avail : n; this.pos = (int)((long)this.pos + skipped); return skipped; } }
As you can see, BufferedInputStream does not ensure the number of bytes needed to skip -- if the skipped bytes exceed the current cache length, it will only skip to the end of the current cache.
Therefore, I thought it jumped to the X position, but in fact it was still where it was - so that's why the stripes came out.
solve
If you know the reason, there are more solutions
- Parameter adjustment method: write 3550 as the parameter above to ensure that it just jumps to the specified position;
- The do while loop ensures that when it is less than the number of skips, it continues to jump forward
do{ n -= inputStream.skip(n); }while(n>0);
- Do not use BufferedInputStream
Directly using the original FileInputStream to read does not have this problem. When skip moves on the final byte stream, it will be true and effective.
Finally, 3550 parameters are adopted. At the same time, do while judgment is also made to prevent possible problems in the future.
summary
As the title says, I'm stupid. Really, I only know InputStream Read, the read length may not be enough, but I don't know that the skip length will not be enough.
The so-called weak foundation will shake the earth and the mountains. It is very important to strengthen learning and strengthen the foundation!