40K sat_lco SOLVED

[jr_walker 10]

Large SAT image (40960 x 40960) in Visitor 3

I started my first ARMA project in October 2010. I wanted to make a very large, very detailed terrain (not the best idea, by-the-way). I was frustrated for a long time because I was determined to have a 1 pixel per meter SAT map

( http://forums.bistudio.com/showthread.php?108939-Visitor-crash-importing-40960-sat_lco&highlight=LARGE+SAT). After months (years?) of struggling, I actually had a dream about how to do it, woke myself up, tried it … and it worked! Of course it is fairly simple, although extremely time consuming. I do not recommend this to anyone who is not fluent with Visitor 3.

The technique can be used to increase the detail of the SAT map for smaller terrains as well (2 pixel/meter for a 20K terrain, 4 pixel/meter for a 10K terrain, 8 pixel/meter for a 5K terrain, etc). The technique can be equally applied to the MASK also, if you have the desire to have a higher resolution terrain mask. (While my project resulted in a 40960x40960 SAT map and a 40960x40960 MASK, my experiments indicated that the SAT and MASK files do not have to be the same size [as long as they are squared]).

[i apologize for the long post following, but I can't figure out how to insert a "spoiler" button]

Tools:

 Patience (You need LOTS of this)

 An image processing program (my early attempts used Imagemagick-which worked, but I later used Adobe Photoshop CS5 (64bit) – and loaded the very critical 64bit SuperPNG plugin (free)

 A minimum of 20GB of available disk space (more is better), for storage of your SAT image, Photoshop paging space, etc.

 Visitor 3

Step 1: Create SAT image

I won’t go into too much detail on how to create a 40960 x 40960 image which is a challenge in itself. Briefly, the resulting SAT image must be less than 2GB in RAM or Visitor will not be able to “Import Satellite + Mask …†(more accurately the size of the SAT image + the size of the MASK image must be less than 2GB in memory). When a 40K x 40K image (use RGB-8bit) is loaded into Photoshop, you can see that it consumes almost 5GB of RAM.

[General note: every step in processing an image as large as 40960x40960 pixels takes a VERY long time]

I can offer one very general requirement (based on using Photoshop) to get the size down: reduce the number of colors in the image by whatever method you prefer, such as modifying the contrast, posterizing, or using other filters – no hard rule here, just try to achieve a balance – the intent is to have as much detail in the image as possible while minimizing the number of different colors in the image.

I can offer one very specific requirement (based on using Photoshop) to get the size down: When you think the image is at a good balance of detail vs. number of colors, change the mode of the image from RGB 8-bit to Index (this will make the image look terrible on screen, but will be fine when processed). After the image has been changed to Index mode (this will take a long while), make no changes to the image but simply change the mode back to RGB 8-bit. When the image has been changed back to RGB 8-bit, make no changes to the image, but simply save the image as a SuperPNG file (options: smaller file, no alpha, no interlacing, no metadata). With any luck, you’ll have a 40Kx40K SAT image that is less than 2GB on disk (mine is actually 2,007, 263 bytes on disk and it processes).

Step 2: Create a mini-SAT image

A) After the 40Kx40K SAT image is saved, reduce the image size to 2560x2560.

B) Save this file using SuperPNG as above and name the file so that you know it is the small version (mine are named sat_lco_40.png [the large version] and sat_lco_25.png [the small version])

Step 3: Create MASK image

The MASK image is much easier to work with. Once you have your mask colors mapped out, make sure the mode is RGB 8-Bit and save the image as a SuperPNG file (options: smaller file, no alpha, no interlacing, no metadata). Follow the naming convention used for the SAT image.

Step 4: Create a mini-MASK image

A) After the 40Kx40K MASK image is saved, reduce the image size to 2560x2560.

B) Save this file using SuperPNG as above and name the file so that you know it is the small version (mine are named mask_lco_40.png [the large version] and mask_lco_25.png [the small version])

Step 5: Import the sat_lco (40) and mask_lco (25) into Visitor

A) In your project folder\source directory, rename the 40K sat_lco file to sat_lco.png

B) Rename the 2.56K mask_lco file to mask_lco.

C) Open your project in Visitor 3 and run Import process, being sure to select these two files.

D) Go see a movie or something ;)

Step 6: Copy the S files out

A) When the Import has completed DO NOT start buldozer. Instead, exit Visitor.

B) Make a directory somewhere with an appropriate name (mine is called Layers_S_40). Open your project folder\data\layers directory. Select all the files with a name that begins S_ (in my case it is 1849 files). Copy these files to the directory you created as the first part of this step.

Step 7: Import the sat_lco (25) and mask_lco (40) into Visitor

A) Rename the sat_lco file in your project folder\source directory back to what they were before the import (i.e. sat_lco_40.png and mask_lco_25.png).

B) Rename the 2.56K sat_lco file (i.e. sat_lco_25.png) to sat_lco.png. Rename the40K mask_lco file (i.e. mask_lco_40.png) to mask_lco.png.

C) Open your project in Visitor 3 and run the Import process, being sure to select these two files.

D) Go get a bite or something ;)

Step 8: Copy the S files back

A) When the Import has completed DO NOT start buldozer. Instead, exit Visitor.

B) Copy the S_ files from the directory you created in Step 6 into your project folder\data\layers directory; overwrite the existing files.

Step 9: See the results

A) Open your project in Visitor 3.

B) Launch buldozer.

C) Go see a movie or something ;),

D) When buldozer completes processing your imports, your terrain will be nicely covered with your high resolution SAT map.

[fraczek 4]

Nice. So you basically make two versions of the project to overcome memory constraints, one with lower res MASK, one with lower res SAT, and merge them together into one high res project, right?

[jr_walker 10]

Not so much two versions of the project, but two versions of the SAT and MASK lco but then essentially did merge them.

[bushlurker 45]

Phew!... Finally...

Finally I get a little time off to give this post the reply it deserves! ;)

WOW!!

Well done JR!!

This is a Major Breakthrough!

AFAIK the biggest sat & mask layer that's been achieved so far was about 28kx28k, so this project blows that completely out of the water!

40K sat image... that's... eeeh... a One point Six BILLION pixel image!!!! :butbut:

SuperPNG plugin, huh? I'll have to take a look at that!

I came up against this "too large to process" image barrier myself recently, but I simply cheated :o , you actually beat the system! heheh! Terrific!

One less amenable problem I also encountered while I was messing with BigStuff was - the dreaded "4096x4096 objects bug" - it exists, verified with 3 separate 4096 test terrains and, its a pain in the A**! :(

Here's what I found out about that, just in case it's useful...

With a 4096 x 4096 heightmap terrain, if you....

* Add an object to the Artificial of Natural objects list in the "normal" way then you CAN select it from the list - click to place it - click it again to highlight it, then you can rotate, drag, etc - all normal, in other words.

* Add an object via "Import Templates" then you CAN choose it from the list and click to place it, but then you CAN'T click to select it - you just end up placing another object. This means you can't "mouse manipulate" the object - rotate, drag, etc.. You CAN "rubberband drag/select" it, and at that point the object properties panel is available via the top menu, and changing eg: location coordinates WILL effectively reposition the object, so the objects ARE there, they DO exist, and appear normally in buldozer, etc, but interacting with them via the Vis 2D display is severely limited.

* Add an object to the Artificial of Natural objects list in the "normal" way, AFTER having "imported templates"... In this situation, the previous "import templates" action seems to screw all subsequent additions... Normal manual adding of objects to the already-populated lists is possible, but the "non-interaction effect" seems to "infect" all subsequent object additions...

However - it seems to be perfectly possible to design small "set locations" on a smaller and better behaved terrain, then simply export lists of all objects - offset their location coordinates in a spreadsheet, then reimport the whole scene to it's "final location" on the Big Map.

It's tricky to make any "post-locationing edits" to such stuff, but it IS a - barely - usable workaround.

Anyway... I guess that's all for now... I'll post more on the 4096 bug if I encounter any more crazy behaviour...

Meantime - congrats once again JR, on a Barrier well and truly Broken!!!

B

[branzik 109]

That´s sound incredible.Great work . Do you have some compare pictures?

[Hatchet_AS 201]

Won't mention how much time I might have spent over the years looking at this issue :rolleyes:

Very well done Sir, and the sharing of the knowledge is very much appreciated. :cool:

[HorribleGoat 1464]

This is awesome info! Well done!

[james2464 176]

Interesting but I thought Photoshop CS6 or 5 had a limit of 30,000px2? How would you edit or create a satellite layer greater than this? For example 90k by 90k

Edited June 14, 2013 by James2464

[jr_walker 10]

@james2462

From http://helpx.adobe.com/photoshop/kb/optimize-performance-photoshop-cs4-cs5.html

"Photoshop supports a maximum file size of 300,000 x 300,000 pixels, except for PDF files, which are limited to 30,000 x 30,000 pixels and 200 x 200 inches."

So 90K by 90K should not be a problem ... but I hope you have monster machine! :) Are you going to apply the technique to F.A.T.A (awesome and inspirational island, BTW)? I'd really like to see someone else give this a shot.

@Bushlurker, thanks for the Kudos and detailed discussion regarding the "4096 bug". I had never experienced the bug until I tried automating placement of vegetation (not long after I read the information your provided - it felt a bit of the Jynx :P ) ... still need to smooth that process, just a matter of time.

Walker

[frankltcol 10]

Great I think you got it. Iìm trying with a 1024 x1024 map 12288 metres but I get this spot colrors http://www.clanbrancaleone.eu/clanab/?p=1266. Can you help me?

Thanks,

Frank

[jr_walker 10]

Hello Frank,

I am not sure I understand your question.

Are you saying you are working with a 1024 height map and your island is 12288 meters x 12288 meters?

Please tell me what is the size of your island in meters and also tell me the size of your sat_lco in pixels.

Thanks,

Walker

[frankltcol 10]

Hi Walker,

Thank you very much for your reply. Terrain,png is 1024 pixel. Terrain grid size is 1024. Each grid is 12 meters. total 12288. Sat_lco in pixels is 5120.

Thanks,

Frank

[bushlurker 45]

Sat_lco in pixels is 5120.

This is the main reason for the "blotchy patches of colour" you're seeing on your Sat Layer... It's VERY low-res!

Generally accepted "normal resolution" for a Sat & Mask layer is 1 pixel per meter... In your case, with a 12288 x 12288 meter terrain, you'd want to use a 12288x12288 pixel Sat Layer.

Currently, with your 5120x5120 pixel Sat image, you're only using about 2.4 meters per pixel resolution... that means each Sat image pixel is being stretched to cover a 2.4x2.4 meter area in-game. That kind of enlargement tends to reduce images to a blotchy, pixellated coverage - just like in your picture...

It also looks from the coastline in your picture that your sat image may not be perfectly aligned with your heightmap... it looks a little... off.

I'd suggest you take a look at that too, but first thing to think about is a higher-res sat layer... try 12288x12288 to start with, it should immediately look a lot better...

Also - this thread is really all about JR's fairly groundbreaking "super high res" satellite layers technique... since your terrain is fairly small, perhaps in the future you could consider such a high-res layer (and you wouldn't need JR's sneaky tricks to do it), but for now it's probably best to aim for a "standard res 1px=1m" until you've got the hang of things...

B

[frankltcol 10]

Hi Bushlurker,

thank you very much for your suggestion! I got a higher res map (12288) and got a beutiful result. Un fortunatly now I get also those lines where grids overlap.

http://www.clanbrancaleone.eu/clanab/wp-content/uploads/2013/07/arma2oa-2013-07-23-10-30-10-560.jpg (389 kB)

Can you suggest me how to avoid it? I tried digfferent options in Visitors (I changed segment size from 512 to 3088 but always get those rows).

Thanks,

Frank

p.s: if I roll back to 5120 map, rows along overlapping areas disappear but quality is poorer.

Edited July 23, 2013 by frankltcol

[bushlurker 45]

Hi again...

Firstly - don't mess with that "Segment Size" parameter... it's a basic engine thing and it "likes" to be 512 if possible.

(You'll like it being only 512 too, later on when you've mastered the basics and you're thinking about more advanced stuff like using more than 4 surfaces ;))

Try resetting all your parameters to default... Sat image size to 12288 - all other parameters in the "calculator" section as default, then "Accept Proposed" - then set a suitable Base Texture Layer size... in your case you should be offered "48m x 48m", so choose that...

Now - go look in your "P:\YourTag\YourProject\Data" folder - you'll find a "Layers" folder containing your current Sat & Mask tiles - delete that folder!

Then - reimport your Sat & Mask in the usual way to regenerate the Layers folder and contents at your new settings size...

* Can I also suggest that if you have any further problems you make a thread of your own? This thread is really all about JR Walkers sneaky oversized Sat layers technique - that's a Cool Thing and, in the future, people will be searching this forum section for info on how to make it work. When they find this thread, they'll be hoping subsequent discussion in the thread contains more relevant hints & tips - not a separate discussion about a different topic, interesting though it may be... :)

If you have a problem or a question and you can't find anything relevant by searching this Terrains forum section, it's perfectly OK to start a new thread about the topic - give it a nice, clear & relevant title, so it turns up for other people in future searches... "Settings for Satellite Image Size" or something would do nicely...

A few months down the line, new beginner guys with the same problem will search and find your thread, read the answers you got and be well pleased they found a solution so easily....

B