[_] examples of user control for a sequence of images
Justin L Mills
jlm at justinfront.net
Fri Nov 2 13:55:26 GMT 2018
Chris The problem is that videos store keyframes and then use fractal compression or similar to store difference from the last frame and so on all the way back to the last keyframe. So to go randomly to any frame you have to load the keyframe then all the difference frames before you can access the random frame, you want to show, this always cause a lag. In flash I used to do this sort of stuff, the trick is to keep videos short and get the video encoded for every frame which is perhaps 10x larger filesize, then once video fully loaded you can just set the cue head so for instance you can then allow the video to move left and right as the mouse moves, or you can do things like setup so that a persons eyes follow the mouse, but that takes even more effort as you may want video to cover all angles. Now another approach is to play the movie and every frame take a snap shot this is sometimes done to apply greenscreen real time, you will find examples where a video is drawn onto one canvas and then every frame the pixels are looped through from 0...x and 0...y and processed. For instance with flash I have grabbed frames from webcam over RMTP link and then draw them as a jigsaw, I did actually similar with html5 canvas and a video but was very heavy. You can see how frames are copied https://github.com/nanjizal/JigsawX/blob/master/jigsawxtargets/hxjs/JigsawDivtastic.hx#L349 But as you can imagine you will struggle to encode faster than actual real time play, so you would have to run the whole movie at more or less run time before you can get all the images so that can be too long often for users to wait with a blank screen. Saving an image every frame is same as processing it - you have to copy every pixel so it's heavy. For instance this is just one frame where I use mandelbrot equation, the cost is not the equation but looping through all the pixels on canvas and writting them. https://rawgit.com/nanjizal/Mandelbrot/master/build/html5/index.html The code to loop through the pixels requires width and height for loops similar to the jigsaw example but because I use Kha and WebGL - to render I can easily move this in 3D axis if you wanted to create some form of fan effect. https://github.com/nanjizal/Mandelbrot/blob/master/src/Mandelbrot.hx#L31 If you want to render all the frames from a video ( after you have spent a few minutes grabbing them ) you can render them in 3D with Canvas I ported this https://nanjizal.github.io/canvasImageTriangle/bin/index.html?update=2 but it might be heavy for lots of images, more useful as fallback for webgl perhaps if your feeling like wasting a lot of time! For 3D render best to use WebGL ( especially Kha ;) ). But WebGL is better for any 3D display and with Kha relatively easy as you can export c++ for mobile app as well as WebGL html5. Now another solution is with FFMPEG or similar you can loop through the video and export as PNG images, then you can put the PNG images into a sprite sheet, once you have a sprite sheet it's very easy to go imediately to any frame. You can create spritesheets also from Flash Animate it has option to export frames to png's and you can normally break apart a video on timeline, but FFMPEG maybe better tool ( just harder ). This example uses two spritesheets ( I created in cs3 flash), and effectively just copies parts of each sheet to an onscreen to create the animation in webgl so two triangles per image, you can do this with Canvas, but I happen to use Kha ( webgl ). Obviously you can easily change it to display the frame index that relates to the x position of mouse on screen. https://nanjizal.github.io/GridSheet/bin/index.html?i=1 The relevant drawing code ( see TrilateralXtra for more recent improved version ). https://github.com/nanjizal/GridSheet/blob/master/src/gridSheet/GridSheet.hx#L114 In this WebGL demo I convert draw commands into triangles and draw them onto a canvas and then animate that in 3D, normally I would use Kha as it's simpler to setup pipelines. https://nanjizal.github.io/TrilateralBazaar/demo/binWebGL/ With Tess2 or PolyK and my Trilateral you can draw what shape image you want in 2D or 3D mapping image to triangles, including images from a spritesheet. My cat: https://nanjizal.github.io/TrilateralBazaar/imageFill/build/html5/index.html with Kha I can easily distort this so you could do some crazy shaders on this. This video covers drawing webcam to screen https://haxe.org/videos/tutorials/kha-tutorial-series/episode-073-video-capture.html with Kha you can just setup 3D pipeline and just draw the image on a 3D plane instead. might be worth doing is getting your video into lots of png's and then import them Blender and then use it with Armory3D. Armory3D exports to Kha WebGL and should give you visual control you can then attempt to add scripts on top. I may have gone all over the place but hopefully you will have more of a feel for limitations, I used to be fairly expert in papervision, away3d ( still viable via OpenFL ) and now moved to Kha so I am sure I can help you if your looking for developer, but you have to understand that video does not actually have all the images you want it only has some keyframes the rest it creates on the fly so random fast access is usually not possible from a standard video, you can only randomly access the encoded keyframes. Spritesheets are probably the way to go. Best Justin On 02/11/2018 11:44, Chris Dawson wrote: > user can scrub back and forth through in some way > (e.g. simple slider control, or drag left<>right, or connected to the > page scroll position) > a bit like a 360 product spin in a way, but more for a sequence > showing a product doing something - e.g. a pile of lego bricks > animated into the final model, or opening a pop-up book from flat to > the 3D 'spread', or a timelapse of a painting from blank paper to > finished work. > > and ideally i'm looking for a component/framework to help build this > sort of thing. but first step is to gather some good examples for the > client.