Building a Remote Desktop in Java

On a whim I wondered: could you build remote desktop sharing in Java?

We use remote desktop tools all the time — Sunlogin, TeamViewer — but few people ever think about how they actually work under the hood. Break it down and it's really just three things: capture the screen, shrink the data, and ship it over the network to the other end for playback. Every one of those steps has performance problems, and Java is at an inherent disadvantage on the very first one. This post records the pitfalls I hit at each of the three stages while tinkering with a Java remote desktop, and what I ended up choosing.

Screen capture is the first hurdle

Of course it's possible — it's all just code. But Java is comparatively slow here. Why? Not because Java code executes slowly, but because Java's built-in screen capture is inefficient. Java provides the Robot class for desktop capture; a 2K-screen JPG comes out around 200 KB, and on my RX 2700X CPU a single capture took a full 30 milliseconds. That's well short of 30 fps, never mind 60. And even if you could hit 60 fps, how would you efficiently push that from Java to every remote client?

A bit more on the mechanics. java.awt.Robot's createScreenCapture goes through AWT's native calls: under the hood it copies the entire screen's pixels from video memory into RAM, then wraps them in a BufferedImage. That process is a synchronous full copy — the higher the resolution, the more data gets copied. One frame of raw RGB on a 2K screen is over ten megabytes, and that memory shuffle alone guarantees it can't be fast. Robot was designed for GUI test automation taking an occasional screenshot, not sustained high-frequency capture.

Some suggest multi-threaded capture to raise the frame rate. I don't think that approach holds up — why? Because it burns far too much CPU. The right move is to find a method that's both efficient and light on CPU.

Multi-threading has another hidden problem: the capture bottleneck is in the system call and the memory copy. Multiple threads fighting over that same channel won't necessarily scale throughput linearly, but CPU usage genuinely doubles. A remote desktop is a long-running background tool — maxing out the host's CPU is a non-starter by itself.

Shipping raw images doesn't work

And transmitting images is a dead end, because each image is too big: 200 KB, and 30 per second is 6,000 KB — a serious drain on bandwidth.

Do the math and it's obvious this road goes nowhere: 6,000 KB per second is close to 50 Mbps of bandwidth. Residential upload can't handle that, let alone one host streaming to multiple remote clients at once. And JPEG compresses each frame independently — even when 99% of the pixels haven't changed between two frames, the whole image gets encoded and sent again, wasting everything on redundant information.

Video streaming is the right path

The best approach is a video stream, using the open h.264 format. There's the even better h.265, but I couldn't find an open-source jar for it — h.265 is still in its licensing-fee era. h.264 records data based on which pixels changed in each frame, rather than storing most of the pixel information of the whole image. So an h.264 stream is far smaller than the sum of transmitted images, and much friendlier to the network.

This is exactly the idea behind inter-frame compression in video codecs: the encoder sends a complete keyframe every so often, and the frames in between record only the delta relative to the previous frame. Desktop content happens to be the ideal case for this — most of the time only the mouse and a small window region are moving while the background sits perfectly still, so inter-frame redundancy is enormous and the compression ratio follows.

Recording the desktop and streaming it to clients sidesteps both the bandwidth cost of images and the capture-rate problem. But with video streaming, real-time transcoding and traffic are no small challenges of their own.

The hard part of real-time transcoding is balancing latency against compute: crank the compression up and encoding time climbs — and nothing kills remote control like the picture lagging behind your hand; turn it down and the bandwidth can't cope. Mature remote desktop software generally resolves this tension with hardware encoding. Catching up in pure Java at the software level is a tall order.

The compromise I landed on

What I ultimately used was run-length encoding, transmitting only the changed data in the RGB bitmap captured by Robot. But when screen pixels change too drastically, RLE's data volume balloons. My conclusion: if you really want to transmit the desktop as images, you have to break through both the efficiency of Java's screen pixel acquisition and minimize the transmitted data size — only then can you maximize the remote desktop's frame rate.

To expand on the approach: compare the current frame's RGB array against the previous frame's pixel by pixel, encode only the changed pixels along with their positions, and send those. The receiving end applies the delta onto its locally cached image. Run-length encoding itself is simple: consecutive identical values are stored as "value + repeat count." On a static screen the delta is nearly zero and it works great. But its weakness is just as clear — the moment the whole screen scrolls or a video plays, almost every pixel changes, the delta degrades into a full frame, and with the encoding overhead stacked on top, the data can end up larger than just sending the raw image.

Pitfalls and caveats

1. Robot capture time scales directly with resolution. Test at a small resolution and the frame rate looks fine; switch to a 2K screen and reality hits immediately. Benchmark at your target resolution.

2. Delta transmission must handle the first frame and packet loss: a delta is meaningless when the receiver has no baseline image, so a full frame must be sent first. Lose one delta frame mid-stream and everything after it stays corrupted — either use reliable transport or periodically force a full-frame refresh as a safety net.

3. Don't overlook the receiver's reconstruction cost. Merging deltas back into a bitmap and painting it to the screen eats CPU too — a weak client will stutter just the same.

Wrapping up

The conclusion from this experiment is fairly clear: for remote desktop in Java, the bottleneck isn't the language — it's the efficiency of the Robot capture path and the cost of pure-software compression. Raw image transmission blows up bandwidth; RLE delta transmission suits static screens but collapses under heavy change; video streaming is the engineering answer, except the pure-Java ecosystem lacks a convenient open-source h.264 implementation. If you just want a toy to prove the concept, delta transmission is enough. To build a real product, you'll need native encoding libraries or hardware encoding.