The world of video playback in applications, especially within game engines like Unity, relies heavily on robust and versatile media player plugins. These plugins act as bridges, translating video codecs and formats into visual experiences we can integrate into interactive environments. However, this process isn’t always seamless. Recently, we encountered a perplexing issue while working with the AVPro Video plugin for Unity, specifically concerning 10-bit video files and the DirectShow API. This investigation highlights the complexities of video codec handling, the limitations of certain APIs, and the potential future role of AI-powered solutions like deep learning deblocking filters in overcoming these challenges.
Our initial problem manifested as a crash – a rather disruptive one at that. When attempting to open and play a 10-bit video file within the Unity editor, and even in standalone builds, the application would crash upon exiting play mode or closing. This occurred regardless of whether we initiated playback or simply tried to access the video file’s metadata after opening it with DirectShow. Interestingly, the issue persisted even with LAV Filters installed – a popular codec pack often recommended for расширенной video format support with DirectShow. We also tested enabling the “10-bit” option within the Windows platform settings of the AVPro plugin, but the crashes continued.
A crucial aspect of this problem was the difficulty in detecting 10-bit videos programmatically. While using MediaFoundation, another video API, there appeared to be no reliable method to identify 10-bit video files before attempting playback. This lack of detection meant no way to gracefully handle potential incompatibilities or warn users. DirectShow, however, offered a glimmer of hope. By examining the video track name obtained through the API, we could identify strings like “yuv420p10le,” which hinted at a 10-bit video format. Our initial strategy was to leverage this DirectShow capability: upon metadata readiness, check the video track name, and if a 10-bit format was detected, halt video loading and display an error message. This workaround, while not ideal, aimed to prevent the crashes.
Unfortunately, even with DirectShow and the ability to detect a 10-bit video, the crash issue persisted. This indicated a deeper incompatibility or bug within the video playback pipeline when dealing with 10-bit codecs in our specific setup. The crash logs proved unhelpful, offering no specific error messages, further complicating the debugging process. The last logged event consistently pointed to MediaPlayerEvent.EventType.Closing, suggesting the crash might be related to the _mediaplayer.Stop() or CloseMedia methods within the AVPro plugin during shutdown.
To provide context, our setup included:
- Unity Version: 2020.2
- AVPro Video Version: AVPro Windows 2.1.0
- Operating System: Windows 10 x64
- Device Model: Desktop PC
The steps to reproduce the crash were straightforward:
- Configure AVPro Video to use DirectShow API with LAV Filters.
- Attempt to open a 10-bit video file (no playback initiation necessary for the crash to occur upon exiting play mode).
- Exit play mode in the Unity editor or close the standalone build.
The provided code snippet (from the original bug report) illustrates the event handling within the AVPro plugin, specifically the OnMediaPlayerEvent function, which captures various media player events, including MetaDataReady where the 10-bit detection was attempted, and Closing, the last event logged before the crash.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.IO;
using RenderHeads.Media.AVProVideo;
using UnityEngine.Events;
using System;
public class AVProManager : MonoBehaviour
{
public static AVProManager _instance;
internal MediaPlayer _mediaPlayer;
internal MediaPath _video;
const int seekTimeDelta = 10000;
Texture2D tex = null;
bool playing;
bool currentMediaWasParsed = false;
bool reset = true;
//For changing media
internal RawImage targetTexture;
bool isOpen = false;
private void Awake()
{
if (_instance != null)
{
Destroy(gameObject);
}
else
{
_instance = this;
}
}
// Start is called before the first frame update
void Start()
{
}
internal void SeekForward()
{
Debug.Log("[AV] Seeking forward ! ");
_mediaPlayer.Control.SeekFast((VideoPlayerManager._instance.curDur / 1000) + (seekTimeDelta / 1000));
}
internal void SeekBackward()
{
Debug.Log("[AV] Seeking backward !");
_mediaPlayer.Control.SeekFast((VideoPlayerManager._instance.curDur / 1000) - (seekTimeDelta / 1000));
}
/*
void OnDisable()
{
if (_video != null)
{
Debug.Log("Disposing video");
_video = null;
}
_mediaPlayer?.Stop();
_mediaPlayer = null;
//_video.ParsedChanged -= OnParsed;
Debug.Log("AVPro OnDisable");
}
*/
//Order of events: StartBuffering->MetaDataReady->FinishedBuffering->Started->FirstFrame->Stalled/Unstalled (lose focus)->Text Cue Changed->Closing->FinishedPlaying (if video finishes)
//10Bit error order: StartBuffer->MetaDataReady->FinishBuffer->Started
public void OnMediaPlayerEvent(MediaPlayer mp, MediaPlayerEvent.EventType et, ErrorCode errorCode)
{
switch (et)
{
case MediaPlayerEvent.EventType.Started:
Debug.Log("Creating texture with height " + _mediaPlayer.Info.GetVideoHeight() + " and width " + _mediaPlayer.Info.GetVideoWidth());
//Change Scale of the texture to match video aspect ratio
VideoPlayerManager._instance.changeScreenSize(_mediaPlayer.Info.GetVideoHeight(), _mediaPlayer.Info.GetVideoWidth());
break;
case MediaPlayerEvent.EventType.FinishedPlaying:
Debug.Log("Media stopped");
onStopped();
break;
case MediaPlayerEvent.EventType.Error:
Debug.Log(errorCode.ToString() + " Error in AVPro " + (ErrorCode)errorCode);
onPlayerError();
VideoPlayerManager._instance._playing = false;
this.isOpen = false;
VideoPlayerManager._instance.stopVideo();
break;
case MediaPlayerEvent.EventType.Closing:
Debug.Log("Media File closed");
if (!VideoPlayerManager._instance._playing && !this.isOpen)
{
Debug.Log("Media File loading problem");
}
VideoPlayerManager._instance._playing = false;
this.isOpen = false;
this.playing = false;
this.reset = true;
this.onStopped();
break;
case MediaPlayerEvent.EventType.FinishedSeeking:
break;
case MediaPlayerEvent.EventType.MetaDataReady:
Debug.Log("Media metadata ready");
Debug.Log(_mediaPlayer.VideoTracks.GetActiveVideoTrack().DisplayName);
if (_mediaPlayer.VideoTracks.GetActiveVideoTrack().DisplayName.Contains("yuv420p10le"))
{
Debug.Log("Detected 10-bit video");
//_mediaPlayer = null;
//return false;
}
break;
case MediaPlayerEvent.EventType.ReadyToPlay:
Debug.Log("Media ready to play");
break;
case MediaPlayerEvent.EventType.Stalled:
break;
case MediaPlayerEvent.EventType.Unstalled:
break;
case MediaPlayerEvent.EventType.TextCueChanged:
break;
case MediaPlayerEvent.EventType.FirstFrameReady:
Debug.Log("First frame is ready, begin playback");
_mediaPlayer.Play();
VideoPlayerManager._instance._playing = true;
this.enableSubtitles();
this.isOpen = true;
break;
default:
Debug.Log((MediaPlayerEvent.EventType)et);
break;
}
}
internal void enableSubtitles()
{
//Enable subtitles
if (_mediaPlayer.TextTracks.GetTextTracks().Count < 1)
{
Debug.Log("No subtitles detected");
}
else
{
Debug.Log("Enabling subs: " + _mediaPlayer.TextTracks.GetTextTracks()[0].DisplayName + " " + _mediaPlayer.TextTracks.GetTextTracks()[0].Language + " " + _mediaPlayer.TextTracks.GetTextTracks()[0].Name);
_mediaPlayer.TextTracks.SetActiveTextTrack(_mediaPlayer.TextTracks.GetTextTracks()[0]);
onPlaying();
}
}
public bool PlayPause(string path = "", bool network = false)
{
//Debug.Log("[VLC] Toggling Play Pause !");
if (_mediaPlayer == null)
{
_mediaPlayer = this.gameObject.GetComponent<renderheads.media.avprovideo.mediaplayer>();
}
//if (_mediaPlayer.IsPlaying)
if (VideoPlayerManager._instance._playing)
{
_mediaPlayer.Pause();
this.onPaused();
Debug.Log("Pause");
VideoPlayerManager._instance._playing = false;
}
else
{
playing = true;
Debug.Log("Play");
if (reset)
{
Debug.Log("Set Media");
if (path != "" && network)
{
// playing remote media
path = path.Replace("[", "%5B").Replace("]", "%5D");
_video = new MediaPath(path, MediaPathType.AbsolutePathOrURL);
//_video.ParsedChanged += OnParsed;
//parseVideoAsync(true);
_mediaPlayer.OpenMedia(_video, true);
//Debug.Log(_mediaPlayer.Media.TrackList(TrackType.Text));
}
else if (path != "" && !network)
{
//Debug.Log("Using local path: " + path);
// playing local media
_video = new MediaPath(path, MediaPathType.AbsolutePathOrURL);
Debug.Log("_video path: " + _video.Path);
//_video.ParsedChanged += OnParsed;
//_video.Parse(MediaParseOptions.ParseLocal);
//parseVideoAsync(false);
this.isOpen = _mediaPlayer.OpenMedia(_video, false);
Debug.Log("Bool isOpen: " + isOpen);
if (!isOpen)
{
Debug.Log("Error opening media");
ExtraScreenManager._instance.showNotification("Failed to open video file. Using " + (Windows.VideoApi)_mediaPlayer.PlatformOptionsWindows.videoApi + " Either not running Win10 or LAVFilter not found? Report bug please.", "ERROR:");
return false;
}
//Debug.Log(_mediaPlayer.Media.TrackList(TrackType.Text));
}
//_video?.Dispose();
//_video = null;
}
Debug.Log("Check If Media is ready to play");
//First media playback is done from FirstFrameIsReadyEvent
if (isOpen && !VideoPlayerManager._instance._playing)
{
Debug.Log("First playback");
_mediaPlayer.Play();
VideoPlayerManager._instance._playing = true;
}
return true;
}
return true;
}
private void onPlayerError()
{
UnityEngine.Debug.Log("AVPro Player error: ");
try
{
if (Directory.Exists(Config._instance.getDownloadRootPath() + "/" + AnimeDetailManager._instance.parseFileName(AnimeDetailManager._instance.res.foldername)))
{
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.Arguments = Config._instance.getDownloadRootPath() + "/" + AnimeDetailManager._instance.parseFileName(AnimeDetailManager._instance.res.foldername);
startInfo.FileName = "explorer.exe";
System.Diagnostics.Process.Start(startInfo);
}
}
catch (Exception ex)
{
UnityEngine.Debug.Log("Failure to open Anime Folder: " + ex.Message);
}
}
private void onTimeChanged()
{
VideoPlayerManager._instance.curDur = (float)(_mediaPlayer.Control.GetCurrentTime() * 1000);
}
private void onStopped()
{
CancelInvoke("onTimeChanged");
VideoPlayerManager._instance._playing = false;
}
private void onPlaying()
{
InvokeRepeating("onTimeChanged", 0.25f, 0.25f);
VideoPlayerManager._instance._playing = true;
this.reset = false;
VideoPlayerManager._instance.totDur = (float)(_mediaPlayer.Info.GetDuration() * 1000);
}
private void onPaused()
{
VideoPlayerManager._instance._playing = false;
Debug.Log("MediaPlayer is paused ? " + _mediaPlayer.Control.IsPaused());
}
public void Stop()
{
Debug.Log("[AVPro] Stopping Player !");
if (_video != null)
{
Debug.Log("Disposing video");
//_video?.Dispose();
//_video = null;
}
playing = false;
_mediaPlayer?.Stop();
reset = true;
this.onStopped();
}
}
While the immediate issue is a bug within the AVPro Video plugin, this scenario highlights broader challenges in video processing. The increasing prevalence of high-bit-depth video formats like 10-bit and beyond aims to deliver richer color fidelity and reduced banding. However, this advancement places greater demands on hardware and software video pipelines. Older APIs or incomplete codec support can struggle, leading to instability and crashes as seen in our investigation.
Looking ahead, solutions to these types of video playback challenges might lie in the realm of AI-powered video processing. Deep learning deblocking filters, for instance, are emerging as powerful tools to enhance video quality and potentially mitigate issues arising from codec limitations or format incompatibilities. While not directly related to the crash bug itself, imagine a future where an intelligent AI filter could dynamically analyze video streams, detect potential decoding issues with complex formats like 10-bit, and apply real-time deblocking or other enhancement techniques to ensure smooth and stable playback.
Furthermore, AI could play a role in intelligent codec management. Deep learning models could be trained to predict codec compatibility issues based on video metadata and system configurations, guiding applications to select optimal playback strategies or suggest necessary codec installations to users. This proactive approach could significantly reduce crashes and improve the overall user experience with diverse video content.
In conclusion, while our investigation started with a specific bug report concerning 10-bit video and DirectShow, it opens a window into the evolving landscape of video processing. As video formats become more complex and demanding, the need for robust and intelligent video playback solutions becomes paramount. While traditional codec packs and API adjustments are essential, the future likely holds a significant role for AI-driven technologies like deep learning deblocking and intelligent codec management to ensure seamless and high-quality video experiences across diverse platforms and formats. Further research and development in these AI-powered video processing techniques could pave the way for overcoming current limitations and unlocking the full potential of advanced video formats for content creators and consumers alike.
