Jenkins: Create a Mac Node

Some of our Unity projects require building on a Mac for the macOS and iOS platforms.  We have most of our build processes automated through Jenkins to build on Windows, but we recently went through the process of setting up a Jenkins node on the Mac to handle the additional platforms.

There is an existing guide, Configuring a macOS Machine as a Jenkins Node by Jishnu Nambiar, that is a very good starting place.  However, we still ran into several issues that this guide does not address, likely because we are working with macOS Sequoia 15.3.2.

Mac User Setup

We will also use Jishnu Nambiar's guide, How to Connect to Your Mac Remotely Using Windows/Linux.  There is one issue we were unable to solve: we had to create a second Mac user as an administrator, because otherwise the SSH login would not succeed due to a low-level block by macOS.  There may be a way around this so we don't need to log in with an administrative user, and we will update this guide if we find it.

In addition to making sure the new user you create has administrator permissions, you also need to enable Remote Login in the same location you enable Remote Management.

Jenkins connects to the Mac via SSH, so permissions run through that system.  In this case, we need to give disk access to the SSH server on the Mac.  Go to System Preferences, then Privacy & Security, then Full Disk Access, and grant the permission to sshd-keygen-wrapper.  You may need to try connecting to the Mac via SSH once and try to make a modification on disk to trigger this.

In general, you can test login issues using an SSH client like PuTTY for Windows.  Point at the IP address of your Mac and start a session, then log in with the new user name and password.  You can also set up a public/private key pair to use for authentication instead of a password, though that is not covered in this post.  For additional debugging of the SSH connection on the Mac side, refer to this StackOverflow answer by Rik Renich.  Turn off the Remote Login setting from above, then run the SSH server on the command line with /usr/sbin/sshd -d.  Note that you can increase the verbosity by adding more -d arguments, so /usr/sbin/sshd -d -d -d will get you verbosity level 3.  Turn Remote Login back on when you're done.

Jenkins Setup

The guide from Jishnu Nambiar covers most of the detail needed here.  In our case, we were using Jenkins 2.479.2.  For this, we needed to make sure Java 17 (or newer) was installed on the Mac so the Jenkins node could operate.

We also found that the safest place to set the Remote Directory for the Jenkins node was under the user directory for the new user we created earlier.  In our case, we made a user called "Jenkins Node", and we set the Remote Directory to /users/jenkinsnode/jenkins.  Drive space can be at a premium, and while this is on the primary drive for the Mac, our actual Unity project is on a separate drive.  We can still access it there for remote builds while keeping the Remote Directory on the primary drive.

When using a separate location for the actual project, make sure that the full path to the location does not have any spaces.  (We had a location at /Volumes/Work Drive/Jenkins, and that failed until we renamed the volume itself to WorkDrive.)  Additionally, make sure the directory that will host the files/directories in use has full permissions for the Jenkins user (chown -R <user name> . and chmod -R 755 .).

Git Setup

For our particular setup, our project was in Git for source control, and used Git LFS, and that required some additional setup to work on the Mac in the multi-user environment.  While Git itself was installed along with the Xcode tools, Git LFS was not.  We needed to install it separately and make it available for all users.  This was not a straightforward process.  We could not use Homebrew installs of Git LFS due to permissions issues between multiple users.

First, access the official Git LFS website to download the binary package.  Make sure to visit the website on the Mac itself, as the site will only present downloads for the OS you are currently using and hide other links.  Unzip the package somewhere (your downloads folder is fine).  Then, using a Terminal window, run the installer: sudo <unzipped location>/install.sh

From here, we need to specially link the LFS extension in with Xcode so that Jenkins can use it.  The basic technique is described in this StackOverflow post by d512.  However, due to new security restrictions by Apple, you can't make this symlink into the Xcode folder under normal circumstances.  First you need to disable System Integrity Protection, then do the symlink, and finally re-enable System Integrity Protection.  Follow this guide to disable and enable SIP.

With SIP disabled, log into your Mac and bring up a Terminal window.  Create the symlink for Git LFS: sudo ln -s /usr/local/bin/git-lfs /Application/Xcode.app/Contents/Developer/usr/libexec/git-core

With that complete, re-enable SIP.  Jenkins should now be able to run Git LFS operations, and you can continue using Git LFS with any user on the Mac.

One last step here: make sure to log in to each user on the Mac and run git lfs install from the Terminal.  This sets up some hooks in Git so that it properly sets up LFS file links when cloning repositories.  If you don't do this, Git LFS will not work properly from a newly-cloned repository.

Cabernet

Cabernet is a 19th century vampire RPG with lots of deep narrative choices.  Will you merge with vampiric society and feed on your neighbors? Or can compassion and love outweigh an eternal thirst?

We helped bring Cabernet to all consoles, including Switch, Playstation, and Xbox.  Plus GoG and EGS on top of that, really.  It's a unique game with a lot of cool story reactivity, and totally worth checking out!

Unity Logging Still Happens in Release

 We recently spent two days chasing down a major hitch, that ultimately boiled down to two things.  First, turns out, Unity doesn't necessarily disable its logging in Release, it silences it.  Second, really tall callstacks take a while to format text for.  Probably.  Best we can tell, anyway.

This was using engine version 2022.3.36f1, for reference.

This started with a major framerate hitch on one of our platforms, on the order of multiple seconds.  This was detectable on other platforms but much less substantial, and measuring it in a profiler we saw LogStringToConsole taking around 30ms in the editor, in the right situation.  That's already hitch-worthy, but it's not usually that bad and obviously much milder than on our problem platform.  So what's going on?

Initially we dismissed this, because in Release logging is disabled, right?  But as we started trying things we started being suspicious, since everything else in the area seemed benign and removing other items didn't really move the needle any.  We had assistance from QA who tested a few previous builds and verified this wasn't entirely new, but it was dependent on how long you played.  Loading into the level and testing it wasn't an issue, but leaving the game to soak for an hour was.

Eventually we got suspicious and removed all the log prints, and the hitch went away, even in Release.  Interesting.  Suggests Unity doesn't disable its log printing, sure, but why would a log print take so long even if that is the case?

Well, this particular project uses coroutines heavily.  It's built partly with Unity's visual scripting and this particular coroutine originates there.  It does its work, then at the end of execution it calls itself, and starts another iteration.  So over time you have a coroutine starting a coroutine starting a coroutine, basically forever.

Perhaps you've noticed, whenever Unity prints a log print it always includes a callstack in the log.  The editor hides this but it's present in the actual log output.  Because of the above setup, that callstack inevitably gets longer and longer, eventually being hundreds or thousands of lines long.  Why is there a gigantic hitch?  Best we can tell, because it's building that massive callstack print and formatting it.  On a slower platform, and depending on how it handles logging, this can be very bad.  Even on PC it's detectable as a few MS that maybe causes a one-frame skip.  On more controlled platforms, it's possibly going through a few layers of operations before it gets to the actual output, and who knows what those are doing

The good news is, you can actually disable logging, by setting Debug.logger.logEnabled to false.  This removed the hitch entirely, including its presence in Release builds.  So this is what we ultimately did.  But it took a long time to figure out this quirk, so here I am documenting it for the public.  Hope this saves somebody else some time.

Sprawl!

A fast-paced, hardcore retro-FPS set in a sprawling megalopolis.  Escape the walled city and take on the militarized government of the sprawl!  Sprawl is an Unreal based title, and we helped bring it to console platforms!

Sprawl is now available for Xbox and PlayStation platforms!

Encoding a Video With Alpha for Unity, Using FFMPEG

It took a surprisingly long time to track down a working combination of arguments for this, so here's one that's known working in an actual project.  This is using Unity version 2022.3.36 and FFMPEG version 7.0.x.  Allowing the asset to re-encode worked fine on all platforms we tried, including all consoles, except the SteamDeck somehow.  We ended up disabling re-encode on PC for that reason.

ffmpeg -framerate 30 -i Frames\Blink\Blink_%%02d.png -b:v 8M -c:v libvpx -pix_fmt yuva420p -metadata:s:v:0 alpha_mode="1" -auto-alt-ref 0 Videos\Blink_VidAlpha.webm

Signalis!

 A classic survival horror experience, Signalis asks you to unravel a dark mystery in the confined spaces of an offworld government facility.  You are Elster, a technician seeking her lost partner and lost dreams.

Although we didn't handle the original port, we were approached to get the game updated with new content from the PC release.  These updates are now live on all console platforms!