Linux

[PluMGMK]

Check this out: https://gist.github.com/thesamesam/2239 ... 78baad9e27 github.com/tukaani-project/xz/issues/92
An attempted circumvention of SSH authentication on Debian and RedHat, via XZUtils (i.e. at this point a very widely-used compression library), has been meticulously put together over several months using several layers of obfuscated code. And it looks like the person putting it together has been the maintainer of XZUtils for years. State-funded I guess?

It's interesting to see the analysis of how the exploit is so deeply hidden in XZUtils, and how it carefully loads itself into SSHD itself under very specific circumstances. Its operation is fascinating, but the way the evil stuff was added to the repository with brazen lies and technobabble in the commit messages is chilling…

(EDIT: The first link I posted no longer works as GitHub disabled the XZ repository in the fallout from all of this)

[Steo]

This is pretty disturbing. I also use Arch at this point so I wonder would that have made it as far as this. Fifo had tried to message me about this yesterday also, but I'd forgotten to look into it.

EDIT: Yikes!

Code: Select all

[steo@steo-arch ~]$ xz --version
xz (XZ Utils) 5.6.1
liblzma 5.6.1

EDIT 2:

Code: Select all

Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:

ldd "$(command -v sshd)

[Fifo]

Fifo had tried to message me about this yesterday also, but I'd forgotten to look into it.

Yeah I wasn’t sure whether you knew about it so I thought I’d letcha know

Code: Select all

Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:

ldd "$(command -v sshd)

Phew at least you’re safe!

Anyway, both my main machine and my VPS have 5.4.x, I checked yesterday

[Steo]

Yeah I wasn’t sure whether you knew about it so I thought I’d letcha know

Thanks for that, it turns out pacman has already gotten rid of the version with the backdoor also.

Phew at least you’re safe!

Anyway, both my main machine and my VPS have 5.4.x, I checked yesterday

Yeah it's good that it never affected Arch, but I have updated in case regardless, as there may even be more to the backdoor that went undiscovered.

Nice also, it's great to not have disease ridden computers.

[PluMGMK]

More interesting info on the backdoor:
A nice writeup that attempts to piece the story together, which covers in some detail the social engineering aspect to this whole débâcle (apparently in train for 2~3 years)
An explanation of how the evil code is hidden in the fake test files, and extracted by obfuscated bash scripts – basically one big "corrupt" file can be decorrupted to turn into a binary object file containing most of the backdoor logic, and a bash script that appends itself to the Makefile and feeds a slightly-altered version of one of the legitimate C source files into the compiler, to make it link to the evil object.
An analysis of the functions and data in the evil object itself, which translates the innocent-looking symbol names in the object to their true purpose, and shows some decompilations (I had started looking at this myself, but this person's gotten a lot further than I have)
List of strings coded in the backdoor – they're encoded in a trie (which I must admit is a thing I hadn't heard of ) to avoid anyone being able to just spot the strings in the binary file; the list contains only SSH stuff, which gives some reassurance that the backdoor wasn't (initially) targeting any other vulnerable software

Honestly, this stuff reminds me of some of the stuff I've worked on, except this person (or group ) had way more advanced skills and (presumably) more resources to dedicate to it – and of course, I only ever use my skills for good, not evil!

It was a really impressive attempt – I'd find it sexy if not for the chilling social engineering and evil intent. It was also damn lucky that Andres Freund noticed that SSH was running slow and followed his nose to expose it when he did – a lot more damage could have been done otherwise. This was a narrowly-averted infosec disaster of epic proportions, and the incident may yet do serious damage to the philosophy of FOSS. I'm actually surprised that I haven't heard about it in the mainstream press yet (I guess we won't, since the disaster was averted and a lot of people mightn't realize how close it came… )

EDIT: See also infographic here: https://infosec.exchange/@fr0gger/112189232773640259

[Reese Riverson]

Since TrueNAS Scale is linux based, I think this is as safe of a place as any here for me to rant a little about my current server issue that I briefly touched on here.

So I'm running a Dell R730XD, which essentially replaced both my Dell R710 and a Supermicro server that share the same generation of Xeon chips as the Dell R710. The Dell R710 was my VMware ESXi box and the Supermicro was my FreeNAS box, and the move to the R730XD was part of a consolidation project of mine. Since I didn't like the direction VMware is going since they were acquired by Broadcom, getting rid of the free versions and jacking up the costs skyhigh, it was only natural to decide between TrueNAS (Formally FreeNAS) and Proxmox. Specifically TrueNAS Scale and Proxmox are linux based, debian I believe. Where as TrueNAS Core is FreeBSD based.

Either way, the virtualization features of TrueNAS Scale and Proxmox interested me since I can still run my storage server all the same. The last unit was purely HDD based, but this new setup consists of both a larger HDD array and an SSD array to rival the old setup.

So here's a breakdown on what I've been running in the Dell R730XD, and why I chose TrueNAS Scale:

• 2 x Intel Xeon E5-2687W v4 Processors
• 192GB DDR4 ECC Memory
• Dell Intel X710 Quad Port 10GbE SFP+ Network Card
• Dell PERC H730P RAID Controller (Configured to HBA mode)
• Dell PowerEdge 12Gbps SAS HBA Controller (IT Mode) - (eSAS)
• Dell PLX PCI-e Switch Card (Connects the 4 x 2.5" U.2 NVMe drives on the front hotswap bay.)
• 2 x Dell EMC WD Ultrastar SS540 800GB Enterprise SAS SSD 2.5"
• 3 x Kioxia PM6-R 15.36TB 2.5" SAS SSD
• 1 x Samsung PM1643a 15.36TB 2.5" SAS SSD
• 4 x Intel Optane 905P 1.5TB 2.5" U.2 NVMe SSD
• NVIDIA Tesla P40 - 24GB GPU (I originally ran an EVGA NVIDIA Titan X - 12GB GPU in here.)

Then I have an HP Enterprise StorageWorks D2600 enclosure connected from basically SAS2 -> SAS3 connection of the HBA controller. Running:

• 6 x 18TB Western Digital 18TB UltraStar DC HC550 HDDs

Since the primary function of this server is to serve as my NAS, I opted for TrueNAS because in order to maintain the 2.5" hotswap bays with the internal SAS controller, for both the SSD array for my storage and mirrored SSDs for the operating system itself, it just made more sense to go this route over Proxmox. Otherwise I would have had to get another storage controller and split the rear 2 x 2.5" SAS drives from the 24 x 2.5" bays up front with PCI-e passthrough being a requirement for getting this stuff working.

The two 800GB SAS SSDs are mirrored for the install of TrueNAS Scale. Each pair of the Intel Optane drives are mirrored and combined into their own pool for VMs. Then each pair of the 15.36TB SSDs are mirrored and put into the same storage pool for a total capacity of 30.72TB of storage capacity.

I then have the HDD array in RAIDZ2 for a 72TB capacity with the ability to lose two drives at once without data loss.

Storage wise, everything's working out nicely, I have it configured to take snapshots of each dataset and replicate anything on the SSDs onto the HDDs, so in the event of anything going wrong with the drives or files, I have backups on hand within the same system. The snapshot feature is especially handy.

The secondary function of this server is to host both Emby and Plex, which is where the GPU comes into play. Then a VM for my Palworld server and other random game serves I want to run for me and my friends.

The tertiary function more or less was running Stable Diffusion, which was done in the same virtual machine I ran Plex and Emby on.

I had zero issues with isolating the Titan X and doing a PCI-e passthrough to the specific VM running Emby, Plex, and Stable Diffusion. The only thing I had to do was apply a NVIDIA driver patch in the VM itself to unlock the artificial limit for the NVENC streams needed for Emby and Plex, and giving the VM access to all 6 CPU cores. Especially for the 4k content.

Then here comes the problem...

I buy the NVIDIA Tesla P40 off my IT buddy, and the past two days of struggle I've yet to get this to work in the VM itself. This loads and works flawlessly under TrueNAS itself, and I even installed an Emby docker container through the available apps on TrueNAS, which is a click of a button to install, and things transcode just fine with GPU acceleration. Though the moment I isolate the GPU and try to pass it into the virtual machine, the VM just won't post, the console remains a black screen like a dead computer. The logs don't show any specific errors either. It just shows the same process as booting up the system as it did before. If I remove the graphics card PCI-e pass through device, the VM boots no problem.

It's been a royal pain in the butt because there have been issues some people ran across with TrueNAS, Proxmox, and even some cases of VMware ESXi. Though what few threads I found on it, answers are scarce. I've decided to expand my search regarding the NVIDIA Tesla cards in general, including the baby P4 model, which one person had the same exact issues I did, who got zero help from anyone but found disabling Ensure Display Device worked for him. Sadly that didn't resolve my issue.

I've even looked into some BIOS settings regarding memory mapping specifics settings, and virtualization stuff is all enabled, which is how the IOMMU grouping and original passthrough of the Titan X simply worked to begin with.

The thing is I know you have to deal with grid licensing with NVIDIA if you are going to use this GPU as a vGPU for multiple virtual machines, but I'm not trying to use it as a virtual GPU, just a direct passthrough. So I can't be sure if something funny is going on there, with the memory allocation, or TrueNAS Scale, some linux issue, or what at this point, but it's been really mind numbing.

[PluMGMK]

Oof, that sounds nasty… I remember something about needing to spoof the hardware if you were passing through a consumer-grade card, since otherwise the NVIDIA Windows driver would detect it was being used in a VM and not allow it, but I guess that shouldn't apply to a Tesla. Also if it's a POST problem it's clearly nothing to do with the Windows driver!

Presumably the card has an Option ROM (or whatever the modern equivalent is ) that is loaded by the VM's firmware, which might be doing some shenanigans. Maybe it's even jumping into an infinite loop because it's detected you doing something NVIDIA don't like (like the first couple of versions of Rayman 1 when they detected you had moved an installation onto different hardware)

Is the VM booting using UEFI or an old-style BIOS?

[Reese Riverson]

Actually Microsoft Windows isn't an issue here, since neither the host OS and the virtual machine are Windows. Basically at the simplest form, you can think of a Debian based system running another one in a virtual machine in this case.

I'm actually not sure about the ROM itself, other than from what I could tell, people has had to disable legacy BIOS options and secure boot while maintaining UEFI, including the video option ROM in the BIOS in order to allow things to work. Which any legacy stuff is off on that server, and I do have UEFI enabled.

The VM is set to use UEFI.

I know there were cases where people had to be particular with these BIOS settings just to allow the main system to boot at all, but given TrueNAS Scale doesn't exactly prioritize virtualization, since it's a storage solution at it's core, there really isn't any advanced settings.

I figure that more advanced configuration could be found in Proxmox, but again this system's design favors running TrueNAS bare bones over Proxmox. I mean I guess I could technically do something more easily if I got a SAS3 disk shelf to run the 2.5" SAS SSDs instead, since my eSAS card can run more than one diskshelf. But I digress...

[PluMGMK]

Yeah, I know Windows isn't part of the issue, I was just thinking out loud about how NVIDIA might be sabotaging your setup…

Well tbh I'm stumped. I don't have any NVIDIA cards in my own inventory anymore so I can't really check anything that might help

[The Jonster]

Finally got around to downloading a Mint and Ubuntu iso and uploading them to Virtual Machines (using VMware for these). I intend to use these for now before moving on to other variations.

[Reese Riverson]

I honestly want to give OpenSuse or Fedora a try again, but if I end up doing something like this, I'm gonna want to run a system with something that will allow me to run my SoundBlaster AE-9, the SFP+ 10gbit Intel network card, and two GPUs, since I'd want to run Windows in a VM still and just pass the GPU through so I'd have a way to still run Windows only applications.

[Steo]

I never actually got passthrough working when I tried it, but that was several years ago, and Linux has improved greatly since that point in time.

Speaking of which, I was very surprised yesterday when I realised Arch was still installed alongside windows on my laptop. I booted it, thinking that Linux only uses my weaker AMD GPU, and was very surprised to find out that it just switches like Windows does. As in, the AMD GPU is used for simple stuff to save power, and the Nvidia is used to render more demanding tasks. I'm really surprised about this, because I never seen this working before. I would always have to just force it to only use the Nvidia GPU and that meant it used more power. That was really the only reason I was still using Windows on the laptop.

[Reese Riverson]

Now you have a good excuse to ditch Windows on that thing!

Sadly, things are looking aa bit painful regarding TrueNAS and the NVIDIA Telsa P40. So what I think I may end up doing is going with a 1U GPU server that I'll get off my IT buddy and let that be the AI server to play with. He'll send a GeForce RTX 3060 with that, and I'll just put my P40 in that chassis when it arrives, and then move the 3060 into my Dell R730XD.

Since the Titan X worked flawlessly in VM use, I shouldn't have issues with the 3060 either, given it's not a weird enterprise level GPU. I'll be able to use the VM again for everything, and technically the 3060 will handle Stable Diffusion a lot better than the Pascal series cards, since the memory requirements are double on those GPUs due to them loading the models in 32-bit. So a 12GB 3060 will handle a ton more than my 12GB Titan.

The 1U server will handle the LLMs that want to eat up memory anyway, and I can host SillyTavern in a VM on the TrueNAS rig for an all in one stop location for AI fun between both machines...

And technically the 3060 will offer more on the NVENC department so... there's that going along with more efficiency.

Oh well, better I figure all this out now and change things up yet again on how I want to handle things.

[Reese Riverson]

For any of you linux users out there, there's been a recent vulnerability discovered with CUPS, which is basically the printing service. A UNIX printing system that's common, and when I looked things up, apparently this isn't the first time a vulnerability was noticed?

https://www.bleepingcomputer.com/news/s ... s-a-catch/

Either way, for anyone who mains linux or runs it for anything, please check your systems and disable this service!

My friend's OrangePi had it, but none of my actual linux server installs had the service running or available.

[PluMGMK]

Just seems like a novel phishing vector, given that you actually have to choose to print to a fake printer to make it happen...

[Steo]

So I'm of course still using Linux as my daily driver (Arch), and I never plan to daily drive anything that isn't Linux in the near future. One thing that was interesting to me is how my cyber security course mentions that Linux is an "OS". It's a kernel before anything else (it hasn't mentioned anything about that).

[Reese Riverson]

I do plan on getting the linux rig assembled when I'm not busy being sick especially. I got the case I wanted in, so it's just a matter of getting it done and re-figuring out what drives I had planned on installing as well.

Thankfully I already have the components, RAM especially for this before all the prices skyrocketed thanks to this AI bubble.

[Steo]

Oh yeah I'm lucky I built my computer last year before this price inflation stuff started happening. GPU prices are just generally always insane at this point, but RAM wasn't that expensive not too long ago.

Hope all goes well with the build. I still just main Linux at this point on my main computer because I prefer it. I can understand some people need Windows if they play certain games with anti-cheat or maybe need proprietary software, but I don't play online competitive like that and there are always alternative programs I can use.