Project

I have recently installed a Samsung Frame TV in our family room (story for another post…). It looks beautiful, works great, and when “off” does a beautiful job of displaying photos as almost art, I really encourage it for everyone to check out. My favorite feature is that when it is in the off-mode of the TV, it can show you any Art you would like, including custom uploaded content. This process for uploading images is manual though, and is fairly clunky. My dream would be for it to be automated, to show some of our great family pictures, and also maybe work as a smart home center, and display other useful information such as today’s weather.

I have always wanted a digital frame which I can fully control, which is part of the reason I bought this TV, and now maybe it can come to full fruition.

Project Goal: Show custom content on the TV when it is off in it’s art mode.

Background

The Frame TV works with a SmartThings Android App that allows you to connect to your TV and control basic functionality similar to a remote. In addition to this basic functionality, it allows you to upload custom Art and put a simple digital Mat around the image. This Art then appears every time the TV is turned off, and is shown for ambience. Below is what it the TV looks like in art mode:

Discovery Phase

SmartThings API

The SmartThings API is “the” Api from Samsung right now, so it seemed like the best place to start my investigation. I know that through previous investigations in this area that it allows you to control things such as turning the TV on and off. This Api lives in the cloud, and everything is controlled directly from there. While I have found that simple TV operations could be done though that interface such as controlling the TV input, I did not find anything related to the “frame functionally”, such as displaying images or being able to control displaying. Even TV state such as on/off does not seem to support the additional knowledge about being in an intermediate state where it is not On, and not technically Off (displaying black screen), but instead showing the Art work. When using the this is a concept, but it does not seem to translate to the Api.

Given all that, let’s investigate my next step. Since the Android app is able to do this, I attempted to reverse engineer the application. I followed instructions online of intercepting android traffic, and tried debugging the mobile app. In doing this it became clear that while the SmartThings API works in the cloud, the mobile application only allowed this “Smart TV” functionality when it was running on the same local network as the TV, and the authentication mechanism was different than that of SmartThings. This and other investigation led me to the idea that the app may be using Simple Service Discovery Protocol along with local API’s for this functionality, and not the SmartThings Api.

Simple Service Discovery Protocol (SSDP)

Through some local investigation I discovered that the Frame TV does support SSDP, and it provides multiple services through that, including discovering the TV in the first place. Through a little bit of probing, below are some of the services I discovered.

• DIAL – Simple API for launching applications on the TV.
• Samsung TV Screen Sharing – REST api for communicating with the device.
• DigitalMediaDevice Interface – Generic Api for playing media.

While DIAL seemed to originally be promising, it only supports launching supported applications (eg. Netflix), and does not support generic content. Further investigation on the DigitalMedialDevice interface showed a AVTransport service which allows playing and controlling arbitrary streaming media. This may be starting to get interesting…

Digital Media Device

I found that the Frame TV supports a Digital Media Device interface which has a AVTransport device that allows you to control media on the TV. This allows you to discover, launch, play and stop content on TV. This supports showing stream content such as a movie, but also supports static content such as a particular image (eg. png/jpg). After some simple testing, and a decent amount of debugging, I was able to send a static image to the TV with the right HTTP/XML content.

Bingo!

While I do not know how to replace the current Art image, I have figured out how to show custom content to the TV. For now, this solution may have to do.

Solution

Now that we have a way to dynamically discover the TV device, and a way to show an image on TV, let’s put it all together into a potential solution.

Below is a high level of that solution:

Image Creation

So, the first challenge is generating content (an image) to show. After weighing the options, I decided that HTML was probably the best mechanism to describe content, and also very to create/debug using any browser. As a first step, I then created a simple html page. From there, I found webkit which is able to load website content, and generate an image from that.

While this would work fine on a system that supports X11 display, my server has no display to use. After a little research, I decided to try Xvfb. Xvbf is a virtual X11 framebuffer, which allows to paint and display items in X11 without having an actual real X11 display.

Below is an outline of the image generation:

• Create HTML page showing content to be displayed on TV. This can include almost anything. For me, this is an image, the local weather, and some other various information.
• Creating a python script that loads the html page in webkit (leveraging Xvbf), and then converts that to a static image.
• Wrap this script as a web app, which returns an image as its output.

SSDP

As mentioned, SSDP can first be leveraged to recognize and discover the device. SSDP works by sending a UDP broadcast packed to the network, and then listening for all SSDP capable devices to respond. This will help us discover the device which we are looking to target, along with the interface that we are planning on using. Some simple python code can accomplish this.

SmartThings

While SSDP/AVTransport can be used to display an image, it unfortunately only works when the TV is turned on. This is where SmartThings comes in, SmartThings can be used to turn the TV on remotely, after which point SSDP can be used. Again, after some SmartThings registration are completed, this control can be accomplished from a little bit of python code calling the right interfaces.

Controller

The last piece, is to put together a controller which coordinates the other components, and runs as needed.

Final Solution

Now, let’s look at all the pieces together (reference diagram above).

• Controller
  • Responsible for launching image on screen and clearing when necessary.
  • Calls SmartThings library for turning the TV on and off, calls SSDP for discovering TV and interface, calls TV interface for displaying content.
• SSDP Library
  • Python library for discovering SSDP devices on the network, and returning their identifiers.
• SmartThings
  • Python wrapper library for calling SmartThings API for a handful of commands.
• AVTransport
  • Python library for calling TV interface XML API to start/stop media.
• HTML Content
  • HTML webpage displaying content to displayed on TV. This is already optimized to look correct for 1920×1080 display (the best possible resolution on the TV).
• Image Creation
  • Python CGI application for generating an Image. This is triggered by just hitting the application. Internally this follows all the steps outlined above to create the right image.

In its first release, the way that this works is that I have a cron job that launches every morning, which launches the controlled to display the content on the TV at a specific time. This is seen for a while, and then 45 minutes later the controller is launched again to clear the image and shut off the TV. This could use many refinements, but is good enough for what I needed in its first version. Below is what this looks like on a typical morning:

Conclusion

While this doesn’t accomplish exactly what I wanted it to do in the manner I was expecting, it does accomplish the overall goals of the original project. I am able to automatically display content on my TV, which updates every day, with almost no interaction from myself.

There was a lot learned during this project about the Samsung TV, different protocols that I wasn’t aware of (ie. SSDP, Digital Media Device standards, etc), and other technologies that existed. While it did not go nearly as smoothly as I had hoped, and took a lot longer than I expected, I am very happy with the results and am excited about how I can extend it further.

I hope that you find this informative, and if you have any of the missing gaps of information, then please share them.