Streaming Frames From Server To Client

[The-Box1]

I've been looking through #288 for help with server side, but everything there is geared for just one screenshot. I'm trying to stream 60 FPS over socket.io but the speed I was hoping for didn't show up. I morphed the getFrameFromMemory or whatever it is into something that manually updates the canvas since for some reason when I try to access the pixels of the canvas from the canvas library all I get is white. Here's the function, I'm just running this then using canvas.toDataURL() and sending that string to the client to render. Is there something I'm missing?

`
async function UpdateCanvas() {

  const frameInProgressVideoOutputLocation = await WasmBoy._getWasmConstant('FRAME_LOCATION');
  const frameInProgressMemory = await WasmBoy._getWasmMemorySection(
    frameInProgressVideoOutputLocation,
    frameInProgressVideoOutputLocation + GAMEBOY_CAMERA_HEIGHT * GAMEBOY_CAMERA_WIDTH * 3 + 1
  );

  for (let y = 0; y < GAMEBOY_CAMERA_HEIGHT; y++) {
    for (let x = 0; x < GAMEBOY_CAMERA_WIDTH; x++) {

      let pixelStart = (y * GAMEBOY_CAMERA_WIDTH + x) * 3;

      var id = ctx.createImageData(1, 1);
      var d = id.data;
      d[0] = frameInProgressMemory[pixelStart + 0];
      d[1] = frameInProgressMemory[pixelStart + 1];
      d[2] = frameInProgressMemory[pixelStart + 2];
      d[3] = 255;
      ctx.putImageData(id, x, y);
    }
  }
}

WasmBoy.config(WasmBoyOptions, canvas).then(async () => {
  WasmBoy.loadROM(new Uint8Array(fs.readFileSync('/home/thebox/Desktop/Website/files/roms/pokemon.gb'))).then(async () => {
    WasmBoy.play();

    setInterval(() => {
      UpdateCanvas();
      io.emit('frame', canvas.toDataURL());
    }, frameTransferRate);
  })
});

[torch2424]

@The-Box1

I've been looking through #288 for help with server side, but everything there is geared for just one screenshot. I'm trying to stream 60 FPS over socket.io but the speed I was hoping for didn't show up.

Oh super awesome! Glad to hear you are doing a server side streaming implementation! Sounds like a lot of fun! 😄 🎉

Yeah we can definitely try to get something performant going!

I morphed the getFrameFromMemory or whatever it is into something that manually updates the canvas since for some reason when I try to access the pixels of the canvas from the canvas library all I get is white.

Huh that's quite odd. I'd want to assume maybe the memory location is wrong, or maybe the rom isn't being loaded correctly? Or maybe play() is only running the first frame? 🤔

Here's the function, I'm just running this then using canvas.toDataURL() and sending that string to the client to render. Is there something I'm missing?

Thanks for sending this along, is this open source on a repo I could pull down and try it out? That way I have more context?

Also, if you are running this on the server, which canvas library are you using? That could also be a possibility in why it isn't working?

But besised all of my questions looking at the code here's some suggestions: 😄

• Take a look at how the library sets up it's canvas for browsers. It seems that on the line: var id = ctx.createImageData(1, 1); You are creating a 1x1 image, when you want a 160x144 (i think) image. So maybe it isn't white, but actually a super small image in the upper left corner?

• Also, concerniing the line var id = ctx.createImageData(1, 1);, it seems like you are making it in a loop, thus you are resetting it every time?

Thus, the resulting code would probably look like:

async function UpdateCanvas() {
  const frameInProgressVideoOutputLocation = await WasmBoy._getWasmConstant('FRAME_LOCATION');
  const frameInProgressMemory = await WasmBoy._getWasmMemorySection(
    frameInProgressVideoOutputLocation,
    frameInProgressVideoOutputLocation + GAMEBOY_CAMERA_HEIGHT * GAMEBOY_CAMERA_WIDTH * 3 + 1
  );

  const id = ctx.createImageData(160, 144);
  const d = id.data;

  for (let y = 0; y < GAMEBOY_CAMERA_HEIGHT; y++) {
    for (let x = 0; x < GAMEBOY_CAMERA_WIDTH; x++) {

      let pixelStart = (y * GAMEBOY_CAMERA_WIDTH + x) * 3;

      var d = id.data;
      d[0] = frameInProgressMemory[pixelStart + 0];
      d[1] = frameInProgressMemory[pixelStart + 1];
      d[2] = frameInProgressMemory[pixelStart + 2];
      d[3] = 255;
    }
  }

  ctx.clearRect(0, 0, 160, 144);
  ctx.putImageData(id, 0, 0);
}

WasmBoy.config(WasmBoyOptions, canvas).then(async () => {
  WasmBoy.loadROM(new Uint8Array(fs.readFileSync('/home/thebox/Desktop/Website/files/roms/pokemon.gb'))).then(async () => {
    WasmBoy.play();

    setInterval(() => {
      UpdateCanvas();
      io.emit('frame', canvas.toDataURL());
    }, frameTransferRate);
  })
});

Let me know if this works, and send me a long a github repo if you have one. Supppeeerrr stoked to help you out with this, and see it once we get it working! Thank you! 😄

[torch2424]

Oh and one more note, just for future reference after stuff starts working:

I think it would be more performant, to just pass along an array with the RGB (no alpha) values down the socket. And then handle all the canvas stuff on the client. That way you have to less processing with the canvas library on the server, and I think it's less overhead in general? The only thing we'd have to check, is if the data URL or the stringified array would be faster to send down 😄

[The-Box1]

Hello!

I don't have a repo for all the files as a majority of them include sensitive information that is too much of a hassle to remove and upload with every change. Though here is the file with everything dealing with WasmBoy: gameboy.js Most of the stuff in there is from the old gameboy library I was using

I'm using the latest version of the Canvas library (2.6.0)

Thanks for the tips! I'll try to implement those.

[torch2424]

@The-Box1

I don't have a repo for all the files as a majority of them include sensitive information that is too much of a hassle to remove and upload with every change.

Oh totally no worries on not making a repo. Just in case though, usually for storing sensitive information for a project, it is reccomended to use enviornment files. And use something like: https://www.npmjs.com/package/dotenv . That way you can upload your code, and have an easy way to swap out keys.

Though here is the file with everything dealing with WasmBoy: gameboy.js Most of the stuff in there is from the old gameboy library I was using

This is great! 😄 Again, no pressure to upload your project 😄

I'm using the latest version of the Canvas library (2.6.0)

I am assuming this canvas library? https://www.npmjs.com/package/canvas

Thanks for the tips! I'll try to implement those.

You are welcome! Glad I can help! I do have one more:

https://gist.github.com/The-Box1/7b1d4377bb4d10cdca476afb8890fb2a#file-gameboy-js-L87

UpdateCanvas() calls async functions, which means they'll be run asynchronously. So you should make UpdateCanvas async as well, and then await it. That way when you emit the canvas data on the next line, it will be run after the canvas is updated 😄

[The-Box1]

Yes, that canvas library.

I can't believe I overlooked that! Thanks for noticing that.

[torch2424]

@The-Box1

Awesome! And no worries! let me know if this works / helps 😄

Super open to helping out more! 👍