IMPROVING COMPILE TIMES – AN EPIC ODYSSEY 3

Scene 3

Last time  we were on the verge of something big … we started compiling but it was slow …

Usually the compilation process is slow for 2 main reasons …assuming that you are already using all cores to  compile.

  1. the compiler frontend which does the prepocessing or
  2. the compiler backend which generates templates and code

For 1. the solution is usually as simple as using precompiled headers.

It seems that cry engine doesn’t use precompiled headers. Ok let’s dig a bit in the waf source code again were we find this little comment.

# Disable PCH files when having UberFiles as they bring the same benefit in this case

#kw[‘pch_name’] = kw[‘pch’]
#del kw[‘pch’]

So let’s take the scientific approach ..

and compile only the cryengine project …without pch

1>  [WAF] ‘build_win_x86_debug’ (‘gamesdk’) finished successfully (2m10.210s)

and with pch…

1>  [WAF] ‘build_win_x86_debug’ (‘gamesdk’) finished successfully (2m21.840s)

So they were right to remove the precompiled headers …

Sill the next step would be to check if the precompiled headers are setup correctly and if there are files that are included in all compilation units then they should be put in the pch.

And this is easily doable with a small lua script and the tlog files that msbuild generates in order to track what files have been read from disk. It’s written in lua …or well sort of …

It uses premake ‘s embeded lua machine to do the job. It was easy to setup and it doesn’t have any dependecies. Just grab it from here 🙂

https://github.com/neo2buha/MostIncludedFile

Be warned though .. be default tlog file are utf16 because Microsoft. Lua doesn’t handle utf16 out of the box so just open the file in a text editor and save it as utf8.

Now …after this little detour let’s get back to business.

Tlog files or tracker files seem to be create trough msbuild. Check out the  cl task

Our problem is that cryengine doesn’t run normal cl task but custom tasks calling waf.exe

NOT a PROBLEM I say … just hope on over to stackoverflow and it seems we can enable something similar.

 <!-- Appended tlog to track custom build events -->
    <WriteLinesToFile File="$(IntDir)$(ProjectName).write.1.tlog"
 Lines="@(CustomBuildStep->'^%(Identity)');
@(CustomBuildStep->MetaData('Outputs')->FullPath()->Distinct())"/>

Let just put this in the custom msbuild targets that cryengine has and VOILA we have this result.

^c:\Data\Engines\CRYENGINE\Solutions\.depproj\Cry3DEngine.vcxproj

AAARRRGHHHG …

This is getting ridiculous … it doesn’t seem to read any c files or headers !!!

Ok …let take a break .. regroup and find another approach.

It seems there are some cmakelists file there so we should be able to generate a normal visual studio project.

cmake1.png

There you go some cmake folder are missing !!! AAARRRRGGGH !!! I’m startig to sound like a pirate. Why would you put just half the files and leave something hanging like this ???

OK it’s official I’m giving up on this road.

But I do have a desktop in the other room and a windows tablet that should be able to compile so let try to put them to work.   It seems cryengine supports Incredibuild

and it even free for up to 4 core

So next time I’ll try to set this up and see how it works !!!

 

Improving Compile Times – An Epic Odyssey 2

Scene 2

Ok so we’re back !

If you haven’t read part 1 i’ll just wait patiently for you to do so.

Now … before reinstalling visual studio which will take a lot of time … let’s try to debug this. We are all programmers and that’s what we do …debug things.

Fortunately  waf comes with source code and while python is new to me it;s all code in the end .. .just written in an exotic language where scoping is done trough tabs ?

# Get root node of Windows SDKs
try:
reg = ‘SOFTWARE\\Wow6432node\\Microsoft\\Microsoft SDKs\\Windows’

It seems that in order to detect sdks it will look for a particular registry key … but I’m missing that key. hmm …maybe the vs installer forgot to add it ? never mind I’ll do this myself .  You had one job installer and I have to do it for you 😛

reg_sdk.png

Now waf seems read the python files from the Code\Tools\waf-1.7.13\crywaflib\ folder so i guess i can debug this a little further (have i told you how much I love tools that come with source code ?)

if verbose:
Logs.info(‘[NOTE] msvc ver %s ‘ % (msvc_version))

manifest_file = os.path.join(path, ‘SDKManifest.xml’)

So it seems i don’t have this file either … But but but why would the installer create the folder for the SDK …copy all the files in it … but forget to copy this manifest and create the registry keys . This is fishy indeed … so let’s look at the installer again .

vs_installer.png

Oh so hidden under Windows and WEB development and under UNIVERSAL windows app I REALLY didn’t select the windows sdk 😦

Thanks installer ! that was fun ! and I also learned a bit about python along the way !

Running configure again it now detects properly the sdk !

cry_waf_5.png Opening up visual studio the solution seems to start compiling

29> [ 2/1012] winrc (win_x86|debug): BinTemp\rc_files\Cry3DEngine.auto_gen.rc -> BinTemp\win_x86_debug\_shared\BinTemp\rc_files\Cry3DEngine.auto_gen.1.res
29> [ 3/1012] cxx (win_x86|debug): BinTemp\uber_files\Cry3DEngine\Cry3DEngine_particles_uber.cpp -> BinTemp\win_x86_debug\_shared\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_particles_uber.cpp.1.obj
29> [ 4/1012] cxx (win_x86|debug): BinTemp\uber_files\Cry3DEngine\Cry3DEngine_main_2_uber.cpp -> BinTemp\win_x86_debug\_shared\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_main_2_uber.cpp.1.obj
29> [ 5/1012] cxx (win_x86|debug): BinTemp\uber_files\Cry3DEngine\Cry3DEngine_CBuffer.cpp -> BinTemp\win_x86_debug\_shared\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_CBuffer.cpp.1.obj

But OMG there are literally A THOUSAND files to compile … on  my  core i3 laptop with 2 core 4 threads and 3GB of Ram this will take forever …

will have to see what we can do about this next time …

Improving Compile Times – An Epic Odyssey

Part 1 – CryEngine

Scene 1

From the dawn of time (or compilers which ever came first) people have struggled to improve compile times. And there is a lot of great advice out there on the Internet and I do love reading about it. Check out this or this well written posts. Surely I’ll not have the time or patience to write such a long and documented posts but I will try to have at least a thorough and scientific approach.

So let just start quickly fork the repo and clone the cryengine repo on the local machine

git clone https://github.com/neo2buha/CRYENGINE.git

 

git_clone_cry.png

Let’s also download the sdks as per instructions and put  in the code folder and run cry_waf

cry_waf.PNG

It generated a solution so let go ahead open it and compile it .

And the first problem of the day is …

29>C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\crtdefs.h(10): fatal error C1083: Cannot open include file: ‘corecrt.h’: No such file or directory

It seems crtdefs.h is trying to include corecrt.h which seems to be available only in the windows 10 sdk and not in the windows 8.1 that i also have installed which seems to have been picked up by waf.

 

Command: ‘C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64_x86\cl.exe /nologo /W3 /WX /MP /Gy /GF /Gm- /fp:fast /Zc:wchar_t /Zc:forScope /GR- /Gd /Od /Ob0 /Oy- /RTC1 /GS /bigobj /showIncludes /MDd /Zi /Zo /Fdc:\Data\Engines\CRYENGINE\BinTemp\win_x86_debug\_shared\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_CBuffer.cpp.1.pdb /Ic:\Data\Engines\CRYENGINE\BinTemp\win_x86_debug\_shared\Code\CryEngine\Cry3DEngine /Ic:\Data\Engines\CRYENGINE\Code\CryEngine\Cry3DEngine /Ic:\Data\Engines\CRYENGINE\Code\SDKs\boost /Ic:\Data\Engines\CRYENGINE\Code\SDKs\yasli /Ic:\Data\Engines\CRYENGINE\Code\Libs\yasli /Ic:\Data\Engines\CRYENGINE\Code\CryEngine\CryCommon /Ic:\Data\Engin

es\CRYENGINE\Code\SDKs\Brofiler /Ic:\Data\Engines\CRYENGINE\Code\SDKs\OpenVR\headers /Ic:\Data\Engines\CRYENGINE\Code\SDKs\lz4\lib /IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include /IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\include /IC:\Program Files (x86)\Windows Kits\8.1\include\um /IC:\Program Files (x86)\Windows Kits\8.1\include\shared /IC:\Program Files (x86)\Windows Kits\8.1\include\winrt /Ic:\Data\Engines\CRYENGINE\Code\CryEngine\CryAction /Ic:\Data\Engines\CRYENGINE\Code\SDKs\GPA\include /D_WIN32 /DCODE_BASE_FOLDER=”c:/Data/Engines/CRYENGINE/Code/” /D_DEBUG /DSWIG_CSHARP_NO_IMCLASS_STATIC_CONSTRUCTOR /D_MT /D_DLL /DUSE_BROFILER /DUSE_OPENVR /DINCLUDE_OPENVR_SDK /DCRY_FEATURE_SCALEFORM_HELPER /D_USRDLL /DEXE_VERSION_INFO_0=1 /DEXE_VERSION_INFO_1=0 /DEXE_VERSION_INFO_2=0 /DEXE_VERSION_INFO_3=0 c:\Data\Engines\CRYENGINE\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_CBuffer.cpp /c /Fo c:\Data\Engines\CRYENGINE\BinTemp\win_x86_debug\_shared\BinTemp\uber_files\Cry3DEngine\Cry3DEngine_CBuffer.cpp.1.obj’
29>

cry_waf_2.PNG

And indeed my suspicions are confirmed … although i have the 10.0.10240.0 sdk installed is not picked up by waf. Running cry_way configure –auto-detect-verbose=True shows additional information. it seems that the sdk path is not in the registry .

cry_waf_3.PNG

ok …let try to reinstall visual studio

vs.png

ARGGGHHHH !!!!!

Ok that’s about all the time for today.
Stay tuned for the next part were we will surley improve compile time 🙂

 

 

Getting started with game development

I get asked this question from time to time … Hi ..I like playing games …how do I go about making them ?

Well my friend … ” i like eating food but I don’t really enjoy preparing it :)” … So first of all think really hard about what you really like to do … If the answer is I like making games then skip the next paragraph and dig right in. If your don’t know exactly what you like doing then you might as well give it a try see if it fits ! But let me tell you this…  if you don’t like sitting around for hours tinkering around with stuff, tearing it down and putting it back up , testing it around, see what you can do better  then starting all over again then programming might not be for you.

IF you are still reading then let me give you some “pointers” 😉 stuff that they don’t teach you in school although they should.

0x0001 VCS

You will need to learn about VCS ( that’s versioning control system )

Read all about it here: https://en.wikipedia.org/wiki/Revision_control

There are quite a few out there … tfs , perforce , svn , git , mercurial … you might as well start with git as it seems to be most popular these days.

0x0002 Learn to google things

Back in my day …you had to put in one of the cds with MSDN and actually read documentation … or wait until late at night when you could connect to the internet and find something on altavista or in some dark chat room … But nowadays it much easier …you look it up on google .. if you can’t find the answer just post the question on stackoverflow and wait for someone with enough experience to give you an answer …

0x0003 Learn to script things …

you’ll find yourself lots of times doing the same thing over and over again… don’t … just don’t … learn a scripting language  and automatize things. it’s much much faster this way.

0x0004 Modify some games

You’ll have to start small ..get  a game that is already working and modify it . There are plenty out there on sourceforge or github. If you don’t want to search any further you can start with a small game that I’m toying with :

https://github.com/neo2buha/videopoker

0x0005 Make your own game

Once you have modified enough games you can start making one from scratch …

Come up with an idea …

Make a plan …

Choose an engine fitted for your game ..there are plenty out there …Unity , Unreal ,  open source ones …

Far this was easy …the hard part is actually finishing a game .. or at least have it playable.

If you have done all these and still like then you might as well make a living out of it.

0x0006 Get Hired

read here some great advice :

https://k0k0k0.files.wordpress.com/2013/05/100-things-every-game-student-should-know.pdf

Once you have set your eyes on a company , check what games they’ve made ..play them … If you can reverse engineer their data format you’ll surely impress someone.

That’s it for now  … stay tuned for the next blog post where I’ll try to compile the game above for android.

Best of Luck!